Hello and welcome to this tutorial this special toy is going to be super exciting because we are getting
closer to the A.S.C. algorithm.
You're going to see that what we're about to implement and that is called eligibility trace or Sarsour
is actually an algorithm of the synchronous active critics agents algorithms that we cannot consider
it.
And we see because we're still going to have one agent but still you're going to see that what we're
about to implement is actually taken from the following paper which is this paper as Synchronoss method
for deep reinforcement learning and it is in this paper that you will find A-3 algorithms that we will
implement as the final bonus of this course.
But as I said we are getting closer to it because the model that will implement right now is actually
this one the synchronous and secularised that's the one that's almost the A3 C which is doing after
that but with one agent and the powerful thing about this is this and step Cunanan we're going to learn
to accumulate rewards and learn the cumulative target on end steps instead of one step like Priestley.
And that's what will make the training much more performance and therefore are much more powerful.
So we actually have the pseudo code for this algorithm it's this algorithm as to right here.
So let's click on it and there we go.
That's the algorithm we about to implement.
But remember with only one agent the difference is that here they take an action.
80 according to the president Greely policy based on the q values for the current state and the action
played.
But in our case we didn't implement an excellent green policy.
We implemented a soft Max.
But the rest is the same as you can see.
We are going to compute the cumulative we worked on and steps actually 10 steps.
Remember the steps is equal to 10.
And so we will implement this line of code in our algorithm that we're about to implement right now.
We're going to get this.
And mostly we are going to implement this as well.
You'll see that we will get the maximum of the q values for the current state and the current action
in this theta.
Here is just a target parameter.
So let's do this let's attack this algorithm.
This one is called the synchronous and stick learning.
But we don't have the right to say synchronous as far as we're concerned because we only have one engine.
But therefore we can call it and learning eligibility trace or even Sarsour.
All right so let's do this.
It's going to be pretty fun.
We can basically follow the code here and that's what we're going to do.
And so as you can see a parameter that we'll need is again up the parameter that is the decay parameter
and therefore we will start by introducing a variable for this Gahanna parameter and choosing them.
So let's do this.
We actually don't need a classroom tremendous We can simply implement this with a function because you
know we don't really need to create objects for this to be to trace model a function will be enough
because basically what we want to do is to return the inputs and the target so that later when training
the AI we are ready to minimize the distance between the predictions and the target and to get the predictions
we need the inputs because we are going to apply our brain on the input to get the output signals that
will be our predictions.
And then once we have our predictions and our targets we will be ready to train the AI by trying to
minimize this great distance between the predictions and the toilets.
So that's the whole point of doing this right now.
We're implementing this function to be able to return these inputs in the Soviets so that we can be
ready for the training to minimize the squared distance predictions manage toilets.
All right so let's do this as we said we want to implement functions.
We start with this.
This we're going to call it eligibility underscored trace.
You can also call it Sarsour.
You can also call it step to ring whatever you want but let's call it eligibility trace and this function
is going to take one argument which is going to be a batch and why that it's because we're going to
get some inputs and some targets because we're going to train the AI on batches.
And so the inputs and the targets will go inside some batches and therefore the input argument here
is this batch that will contain several inputs and then several targets that will compute.
So there we go.
That's the only argument we need.
Now let's go inside a function and let's define what we needed to do.
So as we saw in the Basilica of the paper we need again the parameters so as we said we start by introducing
this gamma parameter semi-close and we can already decipher value.
And we're going to choose four point ninety nine that's a classic good value for the Ghana and Norreys.
I checked that this is a good value for our AI.
All right then next step next step is to prepare our input and our targets because that's exactly what
we want to return.
We want to return the inputs into targets to prepare the training.
And so we can already initialize them with an empty list because of course these inputs Inside the best
we're going to have several input all into a list and that's when initializing the inputs as a list
as well as the targets that we go.
So we initialized our and put in our targets and in the end this eligibility trace function will return
exactly these inputs.
And these yes were of course Filton.
We have several inputs and the associated several targets in what will be returned by the function.
All right next up next step is to start a loop and that's exactly because we're following the slow code
of the paper.
This sort of code.
And as you can see there is this repeat code section and repeated exactly a full loop in the code.
We are going to compute the cumulative reward right here accumulated over the 10 steps.
And how is it computed.
Well in each step that is not the last step we're going to get the maximum of the core values of the
currency that we're in during this and steps run.
And if we reach the last State of the 10 steps well this will be equal to zero.
That is we don't want to do it anymore.
And then we have this for loop which is going to be another up.
They don't say repeat here but that's the same it's going to be the second full loop in our algorithm.
Well we will have at that we were this way by multiplying it by the decay parameter gamma and adding
the word.
So let's do this let's go back to Python and let's start our full so.
And what is going to be the iterative variable.
Well that's going to be our 10 step series.
You know our series of 10 transitions so we're going to call this variable series that represent a series
of 10 transitions like a sequence of 10 transitions.
So for series in.
And then what do you think.
Well our series will be down to our batch.
There is the batches on which will train the AI and so forth series in batch that is for all the series
of 10 transitions in our input batch.
Well where are we going to do well to get a cumulative reward you will see in the silica that we need
the State of the first transition of the series and also the state of the last transition of the series.
So what we have to do right now is get these input states and so we are going to put these two states
into a viable that were going to call input and we will get these two input states.
The first one of the series and the last one that we're going to put into a non-pilot array but no worries
will not stay with this and Ampyra will of course convert that into a horrible but the first step is
to put these two states the first one in the last one into an empire.
And so right here in this array we add the first input which is the input stage of the first transition
of the series and that is Series and then to take it for a transition we take the index zero of the
series thats the first transition and then we can access it by taking its attributes which is state
and thats because in our experience replay file we did find a special structure for each of the transition
and you know the structure.
Each transition is composed of a state an action word.
But then the last element which is done so this special structure that we are allowed to use right now
comes from the way we defined the transition and experience replay.
All right so with this we get the input state of the first transition and now lets get also the input
stage of the last transition of the series.
And to do this that's to saying we can just copy this and paste it and replace as you are here by the
last index of the series which we can access with this trick minus one series minus one that state will
get the input state of the last transition of the series.
All right then we need to put these two elements inside some square brackets because that's what is
expected by the umpire a function and then an important thing to do since we are going to convert that
into a torch answer in a torch variable.
Well remember a torch tensor is by definition a special array containing one single type.
And so we need to force having one single type.
And as usual we're going to choose the float type and so on adding this parameter here D type equals
and P that float.
So that you can take this one and now we can convert that into a torch tensor in a torch voivode.
So let's do this do this.
Well first let's convert that into a torch sensor.
And remember we can use torche that from non-prime that we go and we put all the array of the two input
states inside this torch dancer with the torch from them by function perfect that will convert these
arrays of the two input state into a torch sensor.
And now we put this torch the answer into a torch very well using the variable class so input will be
an object of the valuable class.
And in fact as you understood this variable class takes all this as an argument and that creates the
object.
All right so now we should be good.
We have our two inputs that we need.
That is the input state of the first transition and then input say that the last transition.
And now now that we have the inputs Well what can we get we can get the output signal of the brain of
the AI.
That is the prediction that we're going to call it outputs.
That's the output signal.
And to get the outputs.
Well now that's very easy because we already have a brain created which is our convolutional neural
network.
And so we can simply take our brain CNN applied to the inputs which will return the prediction that
is the output as simple as that.
And now we are already ready to move on to the next step.
And the next step is to start to compute this community if you want.
So now we're going to do exactly the same as our as to algorithm the Sarsour or should we call it and
steps to learning.
We're going to introduce the cumulative reward variable which will be the cumulative reward let's go
back to the paper as you can see right now what we have to do to get this community reward which is
our here.
Well and each step of the 10 steps run we need to update it by adding a zero to this community if we
were if we reached the last stage of the series or the maximum of the core values if we haven't reached
the last stage of the series that is for all the statics that lasted.
So that's simply an bonanzas.
Let's go back to Piscean.
So this community reward as we just saw is going to be equal to zero point zero if we reached the last
state and we can write this condition this way if series of index minus 1.
That is the last transition of the series.
Then we add that done because done actually is an attribute of you know this transitional structure
that we defined in experience we play our experience replay foul and this done comes from actually the
opening structures because if we go to the open Allergan website which is actually right here I prepared
it.
That's the do good or vizir.
And if we go to documentation and then if we.
That's the tutorial I really encourage you to have a look at it.
You can run an environment that Knowsley you can see that our observations that these are transitions
are defined by an observation.
We want this done here and this done means exactly that a transition or a step is over.
And so we're going to use this done here for our IF condition.
Therefore iSeries madness when that done means if the last transition of the series is over is completed.
And so this cumulative reward is going to be equal to zero if the last transition of the series is done.
Else if we haven't reached the last transition well cumulative reward is going to be updated with as
we said the maximum of the key values.
And since this output here is the output of the brain that is the predictions of the neural network.
And as you know the predictions of the neural network are the predicted values Well this output contains
the values.
And since we need to take the max of the q values well we need to add first this index because this
structure contains two key values and the next one.
And then we need to add data to access the data.
This output structure you know it has the special structure of a torch voivode.
So with this we get our core values and then we want to take the maximum of our cue values and so we
add that Max.
And now we get exactly what we want as in the paper this maximum of the cube values for the non-terminal
States as to perfect.
And so now what we're going to do is make the second fold.
That is for the 10 steps of the series we are going to update the cumulative went this way by multiplying
first by Gamma the decay parameter which we already have and then add the B word.
So let's do this.
We are actually going to do exactly the same as in the pseudocode as you can notice they start from
the right.
So they're not starting with the first step and go into the last test they start with the last step.
T-minus 1 to the first step to start.
That's exactly what we're going to do and that's because we want to get in the end the cumulative reward
equal to our equals or zero plus gamma or 1 plus gamma squared or two plus that added plus gamma at
the power of 10 or 10 where are 1 or 2 that are 10 are the word obtained in each of the steps of the
series.
So let's take a quick break before taking the second full loop and I'll see you in the next tutorial.
Until then enjoy AI.