Hello and welcome to the Statoil.
So now the agent has done its exploration and then when he's about to do is to update the shared network.
So the first thing we're going to do is initialize the cumulative what we're going to call it our capital
R and we will initialize it as a torch tensor but that will have dimensions one by one because it's
just a value but we wanted to be a tensor.
And so I'm using here but zeros and then 1 1.
So basically the cumulative reward is initialized to 0.
OK then saying if we're not done that is if the game is not over.
What we want right now is the cumulative reward to be equal to the value of the last trade reached by
the shared network.
So we're going to get the value output.
You know the value of the function outputs of our model and this is the value we will give to the community
we work.
So let's first get this value we can get it this way.
Value Then you know since we only want the value we can add here underscore and then underscore again
and then we get our model because it will output this value but only the first output of the moral thing
to do is double on its course here and here we can just copy paste what we have here.
That is the input of the model with the input images and the pull of the states and the South states.
So I'm just pasting that and there we go.
We will get the value.
And so now what we're going to do is give to our value so all will be equal to value and to access to
the value we at this start there.
All right.
Now the if condition is done and now what we're going to do since we just got a new value by you know
getting the output of the model the first output of the model well that's already append this new value
to the values list.
Therefore we can take directly our values list then towards a tent and we put variable are because our.
This last value so great that is done now.
We are going to initialize the losses and remembering intuition lectures.
You have two losses.
You have the last of the policy that is the last related to the predictions of the agent.
And then you have the last of the value which is less related to the predictions of the critic.
So we are going to introduce these two variables initialized into zero and they're going to take here
policy for us horrible policy loss.
Initialize it to zero and then value lost a lot of the value and say initialized it to zero then let's
not forget to set the cumulative reward as a torch variable because we will need it to be a torch Roybal
because we will be computing a gradient with respect to it because the cumulative reward is going to
be a term of the value loss.
So is this viable it is now attached to the dynamic graphs with a gradient.
And now finally the last thing we need to do before starting the big trend loop you know when we applied
to gas degrade in the sun to reduce this last between the predictions and targets.
Well we need to initialize the GAAP to generalized advantage estimation and not get it or uncoated.
Be careful with that GAAP the variable that we're about to initialize right now is generalized advantage
estimation.
So as a reminder generalized advantage estimation is by definition the advantage of playing the action
a by observing the state s.
So it's a function of the action and the state s and it is equal to the difference between the q values
Q A S and the value of the V function.
So actually I can write it here.
The generalized advantage estimation is a function of the action and the state s and that is equal to
the q values of the action A and the state S minus the value of the V function applied to the state
s.
That's the generalized advantages to mention and that's what we want to initialize right now.
And we will initialize it to zero.
But it has to be towards dancers who were going to use the same trick as what we just did right here
we are going to take the torch library and apply to zebra's function to set it as a tensor of only one
value which is zero.
And we are going to use this new variable g and that will be equal to that torch that zeros 1 one as
initialises us.
So this will be initialized to zero and therefore the q values of the action the state s will be equal
to the value of the V function of the state s.
All right.
And now we are ready to start the for loop.
So we're going to have an adventure here.
So take a good break and I'll see you in the next tutorial to attack that.
And so then I.