Hello and welcome to this tutorial in today's Statoil we are going to synchronize with the shared model.
So what we're going to do is still send the function of course and then initialize the length of one
episode.
So we're going to call the length of an episode episode and this core lengths that we go and we are
going to initialize it to zero.
But then this present length will be incremental.
And speaking of increments in that it's exactly what we'll do.
So we're going to use a while loop and use this trick to say while true Kallen to repeat what's going
to happen now what's going to happen inside this world.
And so the first thing that's going to happen in this loop is this incrementation of the length of an
episode.
So the first thing that we're going to do is incremented by 1 and to do so we can simply take episode
lengths and add here plus equals 1.
And now we are going to synchronize with the share more.
That means that it is now that the agent will get will use the shared model to do its little exploration
on a certain number of steps and how is the model going to get this shared model.
Well we need to take our moral than that and then use the load state Dick's method because we're going
to use it to get the state dictionary of our shared model so we have to put the shared model first and
apply then the static method to get the parameters of the shared model.
And that's how our model here will get the shared model to its little exploration Okay.
And once the model gets this shared model now we have to distinguish two cases.
The first one is if done meaning if the game is done so the game is done then what happens in that case.
Well we have to re-initialize the hidden states and the cell states of the LSD and the mall.
And so that's why I'm going to take See X the cell States and also age X the hidden states and I'm going
to re-initialize them books and how are you going to re-initialize them.
Well with only zeroes there will be a vector of 256 zeroes because remember the outputs of the rest.
As I mentioned 1 and 256.
So there we go we are going to initialize them by using the torch library then the zero's function.
And since we want a vector of 256 zeroes we are going to hear the dimensions one for the vector and
256 for the number of elements which will be zeroes and then we go.
But then we will convert that into a torch Voivode because then some gradients will be computed.
So we need to integrate this with a gradient.
All right.
And we're going to do the same for the hidden states just below and really analyze them the same way.
There we go.
So that's if the game is done.
And now the other case which we can access with Else else than what happens in that case.
Well we're going to keep the old cell States and hidden stakes and so very easily we can keep the old
ones this way by typing see X equals variable c x that data and same for that in the States we can simply
add here H x equals variable x x that data are at.
Good thing done.
Now we can get out of the else because we we're basically done with these two cases whether the game
is over or not but we stay in the while loop because now we're going to do some more things which basically
are all the training process.
And so what we're going to do now is initialize several variables which are going to be at the heart
of the computations in the training.
So let's do this we're going to need the values which remember is the output of the critic.
That's the function.
And we will initialize them as an empty nest.
This way then we're going to need to lug probabilities.
So luck probs and we will also initialize it as an empty list.
Then of course we're going to need all the words that we will also initialize as an empty list.
And finally we're going to need the entropy is something new.
But this is indeed at the heart of the training conditions.
So until just as well.
So now that we initialize these four variables we can start a new FOR loop and then this nymphal will
update the values of these four variables.
And so this new Foluke is going to be a full hoop over the expiration steps and therefore the looping
variable is going to be our steps.
So for a step in range and inside we can directly put perhaps dot non-stops because parameter in some
steps is exactly the number of steps of the acceleration.
So for all the steps in the acceleration What do we do.
Well we're going to get the predictions of the model.
Now what is returned by the model and to get these predictions we can simply take the model and apply
it to the inputs that input signal it goes through the brains in the model.
And that will give us the outputs but it will get several outputs you know it will get us the values
of the function which is the output of the critic.
Then the q values QSA which is the output of the actor but also don't forget that it will also output
the topple of Doheny states and small states because remember if we go back to our model well in the
forward function we can see that indeed it returns the output of the critic.
That is the value of the function yes then the output of the actor which of the cube values QSA and
also the output of the CM which is this double checks and see X then States and the cell States.
So we must be careful with that.
This is quite different than what happened before and therefore we're now going to apply them all to
the inputs which is the state.
But now there are several things to do which are related to torture.
But that gives of course power to what we're doing.
The first thing we need to do is to squeeze in the states to add this dimension that must have the index
0.
That's because the model can only accept a batch of inputs and not an end by itself in a vector or intenser.
That's the first thing we must do and squeeze but then that's not all we need to convert our input states
into a torch voidable so and I in here the very.
So now we are with the state police state but remember that the inputs of the four functions are actually
the input image.
That's what we just took care of but also this type of age X the states and the cell States and therefore
we need to add here.
This second part of the input with its appeal of age X and 6.
All right.
And we must take up the parenthesis.
There we go we have our two inputs the first one is the input states that is input images all converted
into towards variable and squeezed to add this fake dimension of the batch and they stop all of the
states and the South states.
So we're all good to go.
We are ready to get our predictions.
And now since this return.
Well our three predictions the output of the critic the up to the actor and the top all of the reinstates
necessitated by the CME.
Well we're going to introduce some three new variables now which will release three outputs.
So then we get the first output is the value of the V function which is the output of the critic.
So we're going to call it that.
So there we go.
That's the first output.
Then the second output is going to be the output of the actor.
And that's the Q values QSA.
But since the q values are associated to the actions we can also call them the action values.
All right.
And then find an output returned by the morrow.
That's the double of the hidden sales tax and the cell states see X and then we go we have three outputs
returned by them all perfect.
So now that we have the preditions we need to use a soft Max to play the right action.
And so now thats going to be exactly the same as what we did before.
The next step is to get our probabilities so we can call them from and thats where we used to soughed
Max method which we take from the functional module that has a shot at f f that soughed Max and that
will generate a distribution of probabilities of the input that we're about to put right now and which
of course the actual values that is the q values that is the outputs of the actor in the model.
Occasionally we have our probabilities but as you noticed we're going to work with the entropy and together
entropy would not only lead the probabilities but also the LUGG probabilities because the entropy is
the sum of the product Lucke prob. times trub all this multiplied by minus 1.
And so we also need to get our love of which same are going to be generated from LUGG soft max.
So instead of taking a distribution of the probabilities we take a distribution of the probabilities
and that's we do it with LUGG soughed knocks the Optimax function to say we apply to the cube and use
which we call the action values.
All right so now we have the prob and the lock up.
And so we are ready to get the entropy and the entropy.
What is the formula for that.
Well as I just mentioned we take the luck prompt we multiply by the product.
Then we're going to take the sum of all this and to do that we can add here that some one we actually
use the street many times now.
And as we said we multiply this by minus 1.
So it's the minus of the sum of the product.
Lot of times.
Perfect.
And now we are going to store this entropy that was just computed in our list of entropy.
There we go we have the last computation of the entropy.
And so we need to store it in the entropies list and to do this nothing more simple we're going to use
the append function of course because entropies is a list.
So we take our entropies list then start and we use the append function to add the entropy that which
is computed.
All right so we're going to take a break now.
We're going to do this step by step in the next story will play the action by taking a random draw of
this generated distribution of probabilities.
And after we play the action we'll get the value of this state and we will eventually store our new
transition states reward.
And so that will be a new big step done and we will complete that in the next hurdle.
Until then I.