Hello and welcome to the story.
Now we're going to make the for loop that to compute the policy loss and the value loss and once we
have these two lessons we will be able to use our optimizer to place to get cigarette in the sand to
reduce the losses.
All right so there we go.
We start here by the way in the previous tutorial we implemented this section and I forgot to remove
the indents sorry about that.
So starting from here is not in the fall.
And now we're starting a new full loop so I'm starting here with four.
And now what we're going to do is we're going to start from the last step that was done during the exploration
and we're going to move backward in time.
So that's why I'm doing for i in reversed range land rewards biggest rewards is the least.
And since each step of the exploration is associated to where we work because at each step we get reward
when then we words.
Is this number of steps and this reverse here is used so that we can move back in time so that we go.
And now what we're going to do is update the cumulative reward that is far and we're going to update
it this way.
That's actually the same as what we did for Doom.
It's equal to gamma which we get from our parameters and taking from first programs that not times far
plus the reward of this which we can get by taking the least reward and taking the index.
So for us this will be the work of the last and then it will be the reward of the previous day and etc.
and each time we update our By multiplying it by gamma and then adding this reward at the set.
And so by doing this remember we will get in the end.
So I'm going to write it as they come and we will get our community reward that will be cool at the
end of the loop to our zero.
The reward of step zero plus gamma times are one.
We were the first that plus gamma squared times are to the word.
The second step plus that plus gamma at the power of and minus one times the reward attained at step
and minus 1 where any number of steps but then be careful at the end we'll have gamma at the power of
number of steps.
Times devalue the value of the function applied to the last state.
That's what we should get at yet.
And this we will get that because remember here we got this value and the last step because this was
done at the end of this for loop here.
And so we got the value and we set our to be equal to that value.
So right now are at the beginning of the second full loop here will be equal to this value of the last
date.
But then by doing this this is what we'll get in the end are equal or zero percent or one tennis court
or two plus can add the power and minus one times that we were at step and minus one plus game at the
power of number of steps times this value of the Lastings.
So that's the main thing to understand and this can be the action of the cumulative reward.
And that's why it is important to start from it by initialising or with the here and doing this reversed
loop to get this final equation perfect.
And now now that we have the right value for the cumulative reward Well we will compute the advantage
and the advantage here is just the advantage of getting this reward compared to them.
So I'm going to introduce evolvable advantage and therefore it will be equal to this cumulative reward
minus the value of the V function obtained at the stage.
So therefore that is our minus values.
Perfect.
And now that we have the community we work and the advantage then we can get the value loss.
This is the first that we can get now.
So we're going to get our value does very well and this will be updated the following way.
Remember so far that devalued us was initialized to zero.
And so we're going to take the value loss again and at 0.5 times the square to the advantage so we can
get it this way.
Advantage thought out too.
So that just means to square the advantage of the power to and that is exactly the value plus the loss
generated by the predictions of the value of the function outputs by the creek.
And so it makes sense that this is devalued just because remember the advantage of the action in the
state s is the difference between the Q value and the value of the B function.
And so when we play the optimal action Well we get the stationary state with Q optimal of the optimal
action a star player in the state as equals the optimal value.
Vistar of the state s.
So it's quite intuitive to understand that when the advantage is not equal to zero then there will be
a difference between these two.
And therefore that's how the last is measured.
OK.
So very last computed one last down.
We have now one more to go.
It is the policy loss and that is what we're going to compute right now and to compute it we need to
consider again the generalized advantage estimation because to compute the policy loss we need to generalized
advantage estimation and to get the generalized advantages the nation we need first the temporal difference
of the stage valves.
So we have multiple things to compute here and we're going to start with this temporal difference once
we get the temporal difference.
We will get the generalized advantage estimation and once we get the generalized advantages to mention
we will get the peninsulas.
All right.
So let's start with the temporal difference T.G. DD is equal to the reward of the step I plus Ghana
which we get things to our programs list so Bromstad gamma times the value of this debt plus one and
we add that data to access it minus the value of the step I and same we add the data.
All right.
That's the formula of the temporal difference and the state values.
And now we can update the generalized advantage estimation and how is it dated.
Well we take R-GA and we multiply it by gamma parameter gamma times so which we access with our parameters
as well.
So we take programs cell and we add this temporal difference of the state values.
So be careful.
We are in the loop.
And each time we multiply the by and by and we add a temporal difference.
So it's important to understand that at the end of this loop Well this generalized advantage estimation
will be equal to the sum on all the steps of gamma times so that the power of i times the temporal difference
at the step by are so important to keep that in mind.
And now that we have the generalized advantage estimation and the general difference we can finally
compute the policy.
So let's do this.
We are going to update our policy laws the following way by taking the old policy for us and we substract
the LUGG probabilities obtained at the step that we multiply by this generalized advantage estimation
that we have to put in a variable because then we'll compute the gradients.
So it has to be attached to gradients in the graph and then we add minus 0.1 times the entropy.
The entropy obtained at the step in the fall.
And again.
Now be careful.
This is the inside the loop which means that at the end of the flu what you'll get is policy plus equals
minus some over the steps of the product luggin of the policy at the step times to generalized advantage
estimation.
Plus this 0.01 times the entropy does the so that we get.
And now what is the policy of the I.
Well that's the soft Max probabilities of the actions and the entropy of this that I will you know what
it is it's where we computed earlier.
And what we intended to do list.
So we already have that.
But this year I hear it's the soft Max probability of the actions.
And why do we put a minus here.
That's because the luck of the probability and the entropy are negative values.
And since we want to minimize their absolute value we must see this last as the LUGG likelihood as opposed
to a distance.
No we want to maximize the probability of the action that will maximize the advantage.
That's the whole idea behind it.
We want to maximize the probability of playing the action that will maximize the advantage and for those
of you who might be wondering what is the purpose of this entropy efficient.
There is this factor 0.01 here.
Well the purpose of it is just to prevent it from falling too quickly into a trap where we have a distribution
of probabilities with zeros for all the actions except one which has a probability of one.
And if that happens that would minimize the entropy.
So that's why we're adding this small revisions 0.01 year that will make the entropy increase in the
Great in the sense.
OK so now the good news is that the most difficult part is done.
We have the two losses and therefore what we only need to do now and we already know how to do it is
to perform just to get a grade in the sense to reduce these two classes.
And so what we're going to do now is get out of this loop and we're going to take our optimizer.
The one we made separately then remember the first thing we have to do is to initialize all the grading
parameters to zero and to do this we add that then to zero and it's called a grad method.
All right so that's done then.
Now we're going to perform backward propagation but we're going to give twice as much importance to
the policy last than the value lost because the policy is smaller.
So to do this we're going to put in parenthesis policy and the score plus plus 0.5 value loss so 0.5
times the value to us and we're going to add here that we applies the backward method to perform backward
propagation and thanks to this trick here with the policy less plus half of the value that we have twice
as much importance to the policy than the Vaness.
OK then we're going to use another trick which is to prevent the gradient from taking extremely large
values and therefore to generate the algorithm.
And the trick to do that is to get first our torch library then the end and module from the torch library
then the utils submodule and now we're going to use a function CLEP underscore grad's on the score norm
and we are going to input our model parameters with a second input which will be 40.
And that trick will basically make sure that the gradients won't take extremely large values and to
generate the algorithm.
And for those of you who might be wondering whether these 40 years.
Exactly.
Well that just means that we're using these values so that's the norm of the gradient stays between
0 and 40 and therefore that's how we prevent the gradient from taking to large values.
OK now we're almost done.
Remember we made this and sure shared Gretz function at the beginning of the fall which is to ensure
that the agent and the shared model share the same gradients and to do this to make sure that we can
apply this function here.
And so we are going to add and sure share grad's to make sure that the moral and the shared model share
the same gradients.
All right so that's just a precaution.
I'm not sure that's totally necessary but you know at least we won't get an issue here.
Okay.
And finally last line of code we are of course going to perform the optimization step to reduce the
losses and you know how to do it of course we take our optimizer and we add that step with parenthesis
and then we go to training of our brains is over.
So congratulations.
I hope this wasn't too overwhelming.
Don't worry I will provide the code with all the comments.
So if you missed any detail you can have a look at the comments.
And don't worry if you haven't understood anything this is very advanced.
But rest assured this is also the most powerful remember visit made from the creator of pi.
So we are really working with the best here.
The state of the art so it's totally normal if you didn't get everything the first time but by working
on it many times you will definitely get more and more comfortable.
So now we're done with the training.
So basically we made all the most important things you know we made the brains by building the architectures
of the neural networks with the convolutions the LCN and the fully connected layers.
We trained his brain by making this train code here.
So basically the heart of the algorithm is done.
You made the A3 see congratulations.
Now we have a few more things to do but that is just to get the fun part.
You know we need to make this test that we found which will test the agents and provide the videos and
the airplane break out.
So this will be very fun to watch we'll not code all the lines of this test that I fell because as we
said we did the most important thing.
All related 23C but I will of course explain the code and eventually we have this made up I found which
will execute the code.
And from the moment we execute this code all the code will be generated.
So the brains will be made.
The training will happen and the eye will play new games of breakout and we will get all the videos.
So I can't wait to eventually watch them.
We are going to see if he is smart enough to catch the ball.
So now I'm going to see in the next tutorial for this desktop UI so that we can test the AI on some
new games.
And until then enjoy AI.