Hello and welcome to this Python tutorial.
All right.
In this oil we're going to make the function that will select the right action and each time.
So basically we're going to implement the part that will make the car the right move.
And each time that it's going left going straight or going right to reach the goal and to avoid the
obstacles that is descent.
So let's do this right now.
We are going to start as usual with the deaf to define a function and then we give a name to our function
which we're going to call select action than some parenthesis and this select action function will take
two arguments.
The first one is self as you grow to refer to the object and the second argument which according to
you is going to be which one.
Well what could it be.
If you think about it the action we select comes from the output of the neural network because the output
of the neural network or the q values for each of the three possible actions and therefore the action
that we play the action that will be the output of the neural network depends on the input state and
the input states is exactly the second argument we need with select action function.
It's because we're literally going to take the output of the neural network.
And of course the output of the neural network directly depends on the input of the neural network.
So that's going to be our argument.
And now we can give it any name we will actually call it state because the input of the neural networks
are the input states that are and coded by a vector of five dimensions to three signals orientation
and minus orientation.
And so now things are going to be easy.
We are going to feed the input state into the neural network the one that we build right above right
here with the next class and then then we're going to get the outputs which are the key values for each
of the three possible actions and then using the soft Max method which I'm going to explain in this
tutorial.
We're going to get the final action to play.
So let's do this let's go into the function and let's implement all this.
So the first thing we need to start with is about what I've just mentioned soughed Max the idea of the
soft Max is that we're going to try to get the best action to play at each time.
But at the same time we will be exploring the different actions.
And how do we do that.
How can we get the best action to play while still exploring the other actions.
Well we used this idea of stuff which consists of generating a distribution of probabilities for each
of the q values.
Q States action.
Now we have one Q value for each action go left go straight or go right.
But this q value also depends on the input state.
That's exactly the Q function use on intuition lectures.
This Q function is a function of the state and the action.
So since we have here one input state which is the state here and three possible actions we have three
new values Q. State action 1 Q state action 2 and two states action 3 and we are going to generate a
distribution of probabilities with respect to these three key values.
That is we're going to have one probability for the first Q value one of the probability for the second
Q value and a third probability for the third Q And all the three probabilities will sum up to 1.
And so we're going to do all this with soughed Max and soughed Max will attribute a large probability
to the highest Q around.
That's why an alternative to soft Max is a simple RMX no directly taking the maximum of the q values
but in that case we're not exploring the other actions.
Thanks to these probabilities we can explore somewhere else using a temperature parameter that we we're
going to see very quickly.
We can still explore them by configuring this temperature parameter.
That's why in general for security I highly recommend to use a soft x rather than a simple RMX.
All right so let's implements of X and therefore as you understood.
Since soughed Max returns the probabilities of each of the three Q values for the three possible actions.
Well the first variable we are going to create is probably referring of course to these probabilities.
So props equals and now we're going to take our soughed next function and according to you where are
we going to take it from.
Well of course remember we imported the.
And then does functional submodule which I remind is the module that contains most of the actions to
implement a neural network.
We gave it the shortcut F and so that's actually from this functional submodule that we're going to
take our self next function.
But since we gave it a shortcut f we start here with a Neph representing functional from which we take
our soughed next function.
Here it is.
That's the first one and parenthesis.
All right.
Now what do we need to input in the next function.
Well that's of course the entities for which we want to generate a probability distribution.
And what are these entities.
Well these are of course the key values.
So now the question is how can we get the q values.
Well of course the q values are the output of the neural network and to get these outputs of the neural
network.
Well here we go.
We need to take our new network.
But in fact we already have it because that's what initialized in the end it's function.
Know we created a self-taught model which is nothing else that will not work because it is a new object
of the network class.
And so that's perfect.
We can just take our model here and stuff next apply this model to the input state which is the argument
here and that will return the outputs that we're looking for.
That is the key values.
And so now your intuition why we had to take the model here to introduce it in the function might get
better.
Those of you starting with object oriented programming you will see that all this will become natural
so soft next then.
So we take our models self model because this must be the model of the object that we created here.
But then we need to get the output of our neural network model and therefore we're going to hear some
parenthesis in which we are going to input Well the input state named state here.
So what we want to do at first is enter state but now we must be careful to something state looks like
a simple set right now.
But remember that state is actually going to be a torch sensor because later we're going to use this
cell at less state to put it as the argument of the Select action function.
The state's argument that is here is actually going to become later this self-taught less state.
And since this is a tortured answer world the model will accept it.
So that's fine.
But now we can improve the algorithm.
So as soon as the state is a torch sensor and as we said earlier most of the sensors are wrapped into
voivode.
This will also contain a gradient.
So right now what we're going to do first is wrap this input state that is a tensor into a torch very
well but since this is the input states Well there is not going to be some differentiation.
We will not be using the gradient of this state torch Voivode and that can be stations and therefore
what we're going to do now is convert this torch sensor state into a torch variable like so.
But then to specify that we don't want the gradients in the graph at all that can predations at the
end of Mudgal.
Well we will at here come up volatile equals true.
So that now we have our state torched sensor into a torch very well but thanks to this Votel equals
true barometer.
Well we will be including the gradients associated to this input states to the graph of all the conditions
of the end in that model.
So that's another technical trick.
This will save us some memory and therefore this will improve the performance.
So I highly recommend to do this and now we're going to add something more fun.
It's about this temperature parameter that I've just mentioned.
So this temperature parameter is the parameter that would allow us to modulate how the neural network
will be sure of which action it should decide to play.
So this temperature parameter will be a positive number and the closer it is to zero the less sure the
neural network will be when playing in action and the higher the temperature parameter is the more sure
the neural network will be of the action it decides to play.
And to add this parameter I'm going to multiply the outputs which are the Kugan used by this temperature
parameter.
So let's start for example with 7 and I'm going to specify here the little comment T equals 7.
So that's the temperature parameter.
I'm sorry go to 7.
We're going to try some other ones but I just want to start with a small one because you're going to
see that with a small one.
Our car will still behave like some kind of an insect but then by increasing the temperature parameter
our code will look more like a car and decides to sell driving will be much much better.
And so that makes sense because the higher is this temperature parameter the higher will be the probability
of the winning Juval me because for example if we have soft max of the q values.
Let's take some simple numbers one two three if stuffed max of one to three equals.
For example 0.04 0.11 and open eighty five.
Then by increasing the temperature by taking higher temperature.
Right now temperature equals one by taking a high temperature like for example Tussaud subtracts let's
copy this and multiply it by for example two or three so next have the same values but multiplied by
the temperature parameter of three.
Well we will get something like zero for the first Q value because this had a very low probability that
something around zero then something very small for the second probability because this was still a
low probability.
So let's say for example or point 0 2.
But then this third probability since it was the largest one and a pretty high one.
Well increasing the temperature this probability will be even larger because we're going to be even
more sure that this is the right Q value corresponding to the action we must play and therefore this
is going to be something like 0.2 98.
Now by increasing the temperature parameter Well we are now even more sure that the third action here
should be the action to play because the probability for the q value of this action is not only the
largest one but also very high.
So that's what this temperature parameter is all about.
It's about the certainty of which direction we should decide to play.
All right.
So I'm going to remove this comment.
This was just to explain.
And now let's get our action.
So how are you going to do that.
Well the principle of the next method is not only to generate a probability distribution for each of
the key values but also and that's the second step of the soft next method.
We take a random draw from this distribution to get our final action.
And of course we will have a high chance to get the action that corresponds to the Q value that has
the highest probability because that's exactly how the distribution works.
So there we go.
Let's get our action.
So we're going to introduce a new Voivode is we're going to call action and this action is going to
be a random draw of the probability distribution that we just created at this time before.
And so how do we get such a random draw.
Well we're going to take our prop's probabilities of each of the key values we take props and then dart
and then we're going to use the multi Gnomeo function and that will give us a random draw from this
distribution process.
So that's all that will get his reaction.
Perfect.
And now of course we are going to return the action.
There is a little trick here.
What is the fact that this Propst that multinomial returns the PI towards viable with a fake badge.
You know this fake diamonds and corresponding to the batch and therefore to get the right result that
we want that is the action in 0 1 or 2.
We just need to add here data and then some brackets and the actions here are one or two that we're
looking for is content and the index is 0 and 0.
All right.
And there we go.
Now we have our action thanks to this select action function the AI will now know which action to play.
And each time.
Terrific.
So now we can move on to the next function which will be the learn function.
And that's where we will train the whole neural network you know with all the forward propagation and
then the back propagation is is to categorize in the sense.
Well basically we will implement the whole training of the deep learning model that is at the heart
of our artificial intelligence.
So I can't wait to do that.
This is going to be an exciting tutorial and so I'll see you in the next Statoil.
Until then enjoy.
I.