Hello and welcome to this tutorial.
Now we have the brain of tomorrow.
We also have the optimizer.
So basically we are ready to train our different agents.
That is our different brains.
So that's from now that will make this big train function which will contain all the A3 algorithm and
therefore what we're about to implement in this train that I found is just this huge train function
that will just be this big train function nothing else no class but who will use this train function.
And the last step of this module with the main code.
So there we go.
But before we start you can notice that.
Well first we import some libraries so that's the classic libraries with the torch module.
I mean your torch library then ends library to create the environment which will break out.
Then we will of course import are actually class from our model.
File this one.
And finally we will use a variable from TORCIDA.
I regret to run highly performing competitions at the gradient.
Thanks to the dynamic graphs.
And then we have this ensure shared grad's function which I didn't want to spend too much time on this
because well first this is just a function that will make sure everything works correctly.
If the model used by the agent doesn't have any share gradient.
That's why it's called short shared grad's and the other reason is that I don't think this function
is necessary.
But we never know and at least with this will be 100 percent sure that the code will execute properly
but that's not really important.
What we must focus on is this trend function that we all start making right now.
So here we go.
Def and train will soon become a train and this transformation will take several arguments.
The first one is rank.
I'm going to explain what it is and the second the second one is harams so that all the parameters are
the environment.
Then the third parameter is going to be shared moral.
So you know the shared model is what the agent will get to run its little exploration on a certain number
of steps and then finally the last argument is going to be the optimizer that is the one we made earlier.
So perfect that's up for arguments.
And now we are ready to start implementing the same function.
So the first thing we'll do is you know you remember what A-380 stands for it stands for synchronous
active Crilley agents.
So in 8:3 see there is a synchronous.
So as you understood we have to disenfranchise each training agent and to diseconomies then we're going
to use the rank to shift each side with this rank so this rank parameter here is just to shift the seed
so that each training agent is synchronized.
So for example if there is any training agents then the ranks will go from 1 to 10 and there will be
one integer per agent from 1 to 10.
So when we shift the seed by one thread all the pseudo random numbers created by this thread will be
totally independent from the other threads.
However the seed or fixed numbers.
So when we reproduce the experience we will find exactly the same events.
And that's because it's deterministic with respect to the seat.
So it's important to understand that and that's why we need to do is to synchronize each trainee agents
by using the right here to shift the seed with the rank.
So let's do this to do that.
We're going to take our torche library.
Then we're going to get the seed with manual underscores seed parenthesis.
This is a function and now we're going to take the seeds of all the agents which we can access with
from that seed and to shift them by the rank to synchronize.
Each of these agents will just add here plus rec and that will shift the seed with the rank to disenfranchise
each trainee agents because there is one seed for each training agency.
All right first thing done and now next step the next step is to get the environment.
So we're going to create a new variable that we're going to call and and now will use to create Atari
and function from the end module to create the environment for breakout.
That is to get the environment of break out.
So we take this function create Terry and and now we have to input just one argument which are the parameters
of the environment.
And we have them because this is one of the inputs of the brain function.
This is this parameter here which will be the parameters of the environment of breakout and therefore
to get the breakout environments we take these programs argument then that and then we get an name which
in the future that is in the next code with the main function that will execute the whole code will
be breakout brachial Vizier.
All right so that gets us the environment perfect.
And now next step is to align the seat of the environment on the one of the agents.
And why do we do that.
It's because remember each agent of the A-3 Silmaril has its own vision of the environment like its
own copy of the environment and therefore we need to line each of the agents on one specific version
of the environment and to do that we're going to use the seat because each seat determines a specific
environment.
So by associating a different seed to each agent well we'll get exactly what we want that is that each
agent will have its own environment.
And so how can we do that we can take our environment then at that then use the seed function to you
know choose the ones he gets for the environment.
And so now to align the seat of the environment to the seed of the agent.
Well we simply need to get this because this corresponds to the seat of the agent that was shifted things
to rank to get decent organized training agents because they're all on a different set.
So we just need to pay that here and this will align the seat of the environment on the one of the agent
Okay now we are going to get our model.
That is our A-3 see brains.
And so that is now that we're going to use the active class from our model file.
So we're basically going to create a new object of this activity class and we're going to call this
object model or brain if you like.
But basically this object will contain all the convolutions the CM The linear connection and the Ford
function to propagate the signal.
So it will basically contain the brains of the actor in the critic with the ability to propagate the
signal throughout the brain to get the final output.
So let's do this let's create our model so as we said we want to call this object model.
And so we create an object of the Act create class and therefore we take a class actor critic and now
remember what arguments when the two inputs.
That's actually the arguments of the function.
So self we have to input it you know that's what we have to do to use the object in the method.
But then the arguments we have to put are nomine puts which is in bad shape that is done in chains of
art in print images and the actual space that contains you know the set of actions.
So let's put these arguments in the train function.
So the first one we can get it with our environment and that and then we use observation space that's
the space of observations then that and then you get the number of inputs we get shade bracket's zero.
All right.
That's for inputs.
And now for action space.
Well that's almost the same as we need to get it from our environment that we are important than that.
And then action space.
All right.
And that gives us the arguments we need to input when creating an object.
The model of the execrate class.
OK so now we have our model and now the next step is to prepare our input states.
So remember we're still doing deeper informal learning so the input states our input images and therefore
this will be originally done by Ray which will contain one channel because we will work with black and
white images and it will have time in oceans of 42 by 42.
But it's important to understand and to keep in mind here that the input states are input images.
So what we have to do is to get the non-power then we will convert it into a torture answer.
But the first step as what we did previously is to get an umpire and to get it.
It's actually quite simple.
Well first we need to create a variable for the input state which will cross state and this to get an
umpire array.
We simply need to take our environment and then adapt and then use the reset function.
And this will initialize States as an empire array of dimensions one by 42 by 42.
One means 1 channel so black and white image and 42 by 42 is of course the dominations of the image.
The number of pixels and the width and the number of pixels and the height.
So basically that's just the time instance.
And that's the ones we work with.
And now now that we have this actually in umpiring because this will get us these images of such time
insurance in Empire.
Now we can convert them into torch dancers and to do this well we're going to a data state again because
we don't need to keep the number arrays.
And that's where we use torch the torch module.
And remember we already did that to do with the function from underscore non-Thai parenthesis.
And inside this function we need to input the number rate which we want to convert into a torch sensor.
And that is the state the previous version of the state non-pay array will become pipelined from the
pipe function a torch sensor so that just creates intenser from the state.
And now we just need to initialize the done viable.
Remember the variable is generally the variable that says if an episode is over or if the game is over.
Well here we just want to introduce this done very well and initialiser to true to specify that this
Don't Voivode will be equal to true when the game is done.
That will be useful for later so that the AI doesn't play indefinitely to break out.
All right.
So that was basically the beginning of this trend function with some initialization and some things
that we have to do.
The most important part here was that we have to disenfranchise each trainee agents.
That's one first principle of the A3 similar we have to apply.
And now in the next tutorial we will proceed to the synchronization with the shared model.
Let's not forget that there is a different model but also the share model which is a model that all
the agents share.
And so we have to synchronize with the show model so that each agent can get this shared model to proceed
to a small exploration of a certain number of steps.
So that's what we'll do in the next Statoil.
And until then enjoy AI.