Hello and welcome to this Python tutorial.
So let's do this let's make this for loop starting from the right and going to the left and to do this
we're going to add four.
So this is an iterative VAR was going to be our step because we're going to go from the last step to
the first step of a series of transitions and so forth.
And then the trick to go from the right to the left is to use for step in reversed reversed.
And now we just need to input a sequence and this sequence is going to be of course our series.
So we can put our series but as you can see in the paper we go from T minus 1 to start.
So we don't go from the last step that is the terminal stage but the step before that there is to minus
one but to start that is the first step.
And so here to go from not to let's take that step before we need to add in brackets column minus one.
I'm sure that for those of you who followed the machine learning in a deeper course you know this trick
Kollin minus one means that you're going up to the element before the last element but not up to the
last element and therefore we get the sequence we want.
That's what we're going to go from the element before the last element to the first element and that
we do things reversed to go from the right to the left.
All right so we are ready to enter the for loop.
And so inside this for loop what are we going to do.
Where are we going to do exactly as in the paper.
We're going to update the cumulative reward by multiplying it by Ghana and adding the word attained
in the current step that is in the step of the follow up.
All right so let's do this going back to Python.
And so we want to update our cumulated we work the following way by first multiplying its by gamma.
There we go.
Here we multiply it by gamma and then we want to add the reward of that which we can access this way
with the special structure.
Remember that word is an attribute of the object.
And so here of course we add a plus.
All right it's a cumulative reward.
Equals do we want this step.
We are in right now the loop plus Gahanna times the previous cumulative reward before it is a.
Perfect.
So now I think we're good.
We're following thoroughly the algorithm.
And now time for the next steps.
Well now it's going to become pretty easy.
We go back to the first follow up because this for loop is just to compute the cumulative reward no
going from the right to the left by updating this way.
Following the algorithm and now as you remember the goal of doing this is to get our inputs ready and
our targets ready so that we can minimize the squared difference between the two for the training.
And so right now the only thing that we have to do that is get these inputs and toilet's ready.
So let's do this first what we need to do is add the first date of the series in our parts list.
So far this state is in this in part of our goal.
That was just to compute the output.
So we're going to get this input state of the first step separately because that's exactly what we need
to happen in our list.
So let's get this separately.
Therefore we're going to call it state.
And so exactly the same as here we can get it this way by taking the first index of the series which
contains the first transition and then adding that state to get the state of this first transition.
So that's the site we need then saying we're going to get separately the targets associated to this
input stage of the transition.
And so an interesting new variable here target which will be equal to the value of the first step.
And since the Q value is returned by the neural network and its content and output and since outputs
is the output associated to this input which contains the first of the transition well we can get this
q value of the first date by just taking output here and taking the index zero.
And then we add that data that will simply get us the Q value of the input state of the first transition
and that is exactly the time.
Q So that way we take it then we are going to update this target variable but only for the action that
was selected in the first step of the series and to access this first step of the series.
Well we need to take first series 0 because this is exactly the first step at a series 3 0.
And to access the action corresponding to this first step of the series well we need to add here that
action again that is this attribute structure that we're using you know action is an attribute of the
first step of the series.
That is the first transition of the series because each transition of the series has the following structure
state action word and done so action here this attribute action here means that we're simply getting
the action of this first date.
And so the target for that specific action of the first step is exactly what needs to be updated by
the community of the world.
So basically here we're just going to write that target associated to the action that was played.
The first step of the series is this cumulative reward that we just computed.
All right and now we're finally ready to update our input by appending this first stay here and this
first are here for the first hour.
We only need to update the first step of the series because you know we train the AI on 10 steps and
therefore the input is the first step of the ten steps.
And also we get the target in this first step but then we don't get any inputs or any toilets in the
following steps of the 10 steps because basically the learning happens 10 steps afterwards.
Thats why right now we only getting the state and the target of the first step of the series.
So its important to understand that and therefore if we understand that now we understand that we have
to input them in our list of inputs and our list of targets.
So let's do this.
First let's append the states to our inputs.
So we take our inputs list and we use the append function to add the state which remembrancer the input
state of the first step of the series and then we are going to append the target at the first step to
our list of targets and to do this we take our list of targets and say we use the append function to
append this first target.
There we go.
Almost done and now we need to return the last things which are of course what we needed to as we said
at the beginning of this tutorial the inputs and the targets that are now updated.
So we are going to add here every turn and we're going to get our inputs first but then that's the thing
we need to convert them into a number array first then do a type conversion to make sure we have a single
type with the type you equals and that floats 32 the same.
And then we convert this into a torch tensor because of course we're working with a torch that's totally
compulsory.
And so I'm using the torch from non-Thai function again.
And that gives us our inputs.
Perfect.
And now let's do the same for the targets.
Now we can use this trick which is quicker.
We're going to stack the targets together and to do this we need to take first our torch library because
we are going to use the stack function by torch to stack the targets.
All right.
And so this line of code basically returns the inputs and the toilets that were just updated through
this eligibility trace Sarsour algorithm.
Or we can call it and step in and send our congratulations.
We were ready to do the final training because basically the training consists of minimizing the square
differences between the predictions of our inputs and the toilets.
So let's get our eyes smart.
It will become smart and the next its.
And so until then enjoy AI.