Hello and welcome to this Python tutorial.
All right so today we will be making the update function which will update everything there is to date
as soon as the eye reaches a new state.
So when it reaches new stage you know we need to update the action.
The last actually comes the new action that was just played.
But also the last date that becomes the new state.
And finally the last word that becomes the new word we get when we play the action.
So that's the logical path that happens right after selecting an action.
We need to update all the elements of the transitions.
And of course you'll get a new transition.
So we will have to append this new transition to the memory.
And finally we will also update or reward window you know to keep an eye on the evolution of how the
training is going and how the exploration is going.
But what's most important for you to understand is that now we can finally make a connection between
the AI that we're implementing right now to our map because if we go back to our map remember there
is this Beiji update function into the game class and that's where we were actually making the game
with the car and defining how the car should be punished when it's making a mistake.
But in this game class we noticed this update function and in this update function we noticed this line
action because brain update last we worked less signal.
And actually this is exactly what we're about to make.
We are about to make this update function that will take the last word and the last signal to get the
next action to play.
So not only will the update all the different elements of the transition.
But mostly we will be playing the action that we should play when getting the last word and the last
signal and so of course in this update function we will use the select action function that we just
implemented before we will integrate the select action function in the future update function that we're
about to make to select direction to play besides making all the updates.
So that's really important to make this connection with the map right now.
What we're about to make is eventually the connection between our eye and the game the game that we
make in this class.
And so what we can do now is directly take this update.
Last we were less signal because that is actually the function that we'll be making with these two arguments
here.
And just as a quick reminder brain is our AI object that is it's the object of the Dejuan class.
So what we're going to do now is we're going to copy this update less word less signal and that's going
to be our next function we're making.
And therefore and pasting that here then just to be careful I would just like to give some different
names than the names we have here.
You know we have last word here and I don't want to confuse this last word with this one.
That can be dangerous.
So I'm going to replace last word here by word.
And by the way same for less signal let's just put signal or even new signal to specify that you know
we want to measure that when reaching a new state and therefore getting a new signal.
But then of course this word here is going to be the last reward that we get here.
You know when going onto some sand or worse getting too close to one edge of the map that's where we
define the last word.
This last word is going to be the input of the update function so that's why we have last word here.
But right here I'm just giving another name for the argument.
We want to not confuse it with last word here.
All right so this is the update function.
And now let's go inside it and let's do these two things.
That is the data all the elements of our transition and of course select the action.
OK so what do we need to update first.
Well as you understood we want to make a date when reaching any state.
So the first thing we'll be updating is obviously this news that that is the news that we we're reaching.
So I'm going to call this new state new state and then it will.
And so how can we get this new state.
Well of course that depends on the signal the new signal that the sensors just detected.
And as a reminder the state is the signal itself composed of the three signals that the sensors signal
one signal to signal three plus orientation and minus orientation.
That's our state.
So be sure to understand that the signal is the state.
But right now it is a simple list of five elements.
And since this is going to be the input to do all that work remember we have to convert it into a torch
sensor.
So that's exactly what we're going to do right now.
We are going to take our torch library and then take the tensor class there we go which will convert
our new signal into a torch denser then it's better to make sure that all the elements of the torch
to answer are floats so I'm going to make a type conversion to convert them into floats like this.
And then finally try to get a reflex of what we need to do next.
It's of course to create that fake down engine to add diamonds and correspond to the batch and we do
this of course with the squeeze function to which we have to put the index of this big diamond.
And we want to for the batch which is zero.
All right.
And now we have our new states composed of the three signals of the three sensors plus orientation minus
orientation and of course that will depend on the new signal we are getting with this update function.
Right at this time last signal we get the three signals.
So in addition minus or in addition and as a reminder the three signals are the density of sensor detected
around the sensors.
All right.
So we just got a new stage so that means we reached the new stage and now we have to make the next update.
So according to you what do we need to update now.
What would be the logical thing to date right now after reaching this new state.
Well what we need to date now is the memory.
Why is that.
It's because at each time t a transition is composed of the current state Estey the next day as tipis
when the reward arti and the action 80.
And right now we already have S-T we already have our team and we already have 80 and we just got the
last element of the transition Estep plus one.
So by getting this new state as deep as one we are getting one brand new transition of the memory and
therefore we have to append this brand new transition to the memory because that's simply our next transition.
So that's why we have to that memory right now.
And therefore what I'm going to do is take my memory object created from the replay memory class and
therefore I'm going to take self memory to refer to the object.
But since I'm using self I have to include the self in the of that function.
So now you can really see what this self is for.
It's whenever you use one variable that you created and initialized in the init function a that memory.
And now we did it.
And according to you how we're going to update that.
Well the good news is that we already made a function to do that.
It's the push function which bans an event or a transition to the memory.
So that's exactly what we're going to use now.
We're going to use the push function to pen our new transition that we just made to the memory and therefore
here I'm taking not an equal because we're going to use the method and therefore we can directly use
that push.
And first I'm going to add the transition this new transition that we just got and that is first the
last date.
So self that last date.
So that's S-T That's exactly this one it already exists.
Then the next element of this transition is of course the new state that we just reached.
And therefore since it is not viable of the object that we created and initialized in this init function
we don't put itself here.
We directly put the Newstead then the next element of the transition is the action and say we already
have the last section which is this self that last action here.
So of course it is equal to zero.
But then of course it will be updated with the select action function.
But that's this one.
So then it is self that last action.
But now be careful.
The elements that were included in this transition should all be torched answers.
As you can see that's the case for the last date.
It's a torch sensor.
The new state is also torched answer.
And so this must be the same for the action and then the reward of course.
But now you're going to think how can it be a torched answer considering that it's simply a number.
You know the action is either 0 1 or 2.
But in fact that's not a problem.
We can still convert this 0 1 or 2 variable into a torch sensor.
This will just be what we call a long tensor.
The long is a type and that's the tensor that will contain an integer because the last action is an
integer it is 0 1 or 2.
So what we're going to take now is our library torche then we're going to take the long is long tensor
class that will create an object which will be the non-tenure itself.
And by taking this self last action function as input it will create this long tensor object but it
will still contain 0 1 or 2 into a long tensor object and that is just to be consistent with the transition
that it only contains tensors because we're working with PI torch and we're working with a neural network.
So we have to work with sensors so that we go torche long to answer.
And one last conversion to make.
We must be sure that what's inside this long answer is an integer and to make sure of it even if we
already know that the last action is 0 1 or 2 to make sure that we are going to make this int type conversion
again we convert our self last action into an integer.
There we go and then we must just put that integer selfless action into brackets right here so that
now we get a long tensor of one element which will be this last action 0 or 1 or 2 itself.
So the key point is that's just how you convert a simple number zero one or two into a tensor with torche.
All right.
And then finally the last element of the transition and that's of course the last word we got.
That's exactly the last word volleyball we created in any function that was initialized to zero.
But then of course is updated right here in this code either when we go on to some sense which is negative
word or if we get further away from the goal that's again a negative reward.
If we get closer to the goal that's a positive reward and the worst punishment if we get too close to
one edge of the map Well that's a terrible negative word minus one.
And that's a.
So let's add this last element of the transition self-direct the word.
So I'm copying this pasting here and now we have to make another conversion which will be of course
exactly the same as this one only since the word is not an integer but a float number.
We will simply make a torch that tensor conversion but without that it will keep the brackets here because
you know first we have to put the number into a list and then this list will go and put the torch into
to class but we don't have to make that in conversion because last word is a float number.
So what we're going to do is simply add here torche dot tensor torched a tensor then parenthesis brackets.
And we are going to close the brackets here and we close the parenthesis.
There we go.
So to summarize Which did you say that we just reached and the we word we observe a new event of transition
that we add to the memory.
And this transition contains the last date and see the new state as tipis when the last action plays
80 and the last word Archie.
All right.
And now we are going with our memory of dates.
So let's have a quick break and we'll take care of the next update in the next tutorial.
Until then I.