Hello and welcome to the super exciting part of our creation the part where we make it smart.
So that's exactly what happens when training the day I will train it's intelligence to reach the goal
we wanted to accomplish that is reaching the vest without getting killed.
And to do this we're going to basically train the neural network to output the right predictions and
then everything is already ready because these output signals from the brain already have the right
transmission to the body to play the final actions.
So basically now what we're about to do is something we already did before.
We're just going to take some random batches from the memory get our inputs from the samples get the
outputs get the target get the predictions compute the last area between the predictions and the target
and then perform backward propagation to categorize in the sense that the weights according to how much
they contributed to this last error.
So let's do all this.
You're going to see how it's going to be so easy because we already have all the tools to implement
this not only we have the torch tools like the optimizer and the last functions but also we have all
the classes that we made before like our brain of course which we're going to use to get the predictions.
Then our experience replay implementation eligibility trace and all these tools combined to the parts
which tools will make the training super performance and therefore eventually we will get a super powerful
AI.
So let's make this happen.
Let's make our smart and the first thing we're going to do now is get the last function that we're used
to in the training when computing the error and an optimizer.
That's the first thing we'll do.
So let's create a variable for the last function.
We're going to call it loss and this will be equal to the MSEE loss function from the end module.
And that MSEE us that's the last function we'll use because basically our predictions are key values
you know we're predicting the core values of the different actions.
And therefore since these are real numbers well we're kind of doing some neural net work for regression
and therefore the last function is the means CT error.
That's the last function we use.
In general for regression.
All right so now that we have our last function Let's get our optimizer.
So optimized here that's the Voivode we create Freda's optimizer.
And we're going to take as usual.
As for the self-driving car the atom atomizer.
That's a very powerful optimizer that will work wonders for the training.
So let's get this one up in that Adam.
And remember that's exactly what the self-driving car we have to input two essential arguments.
The first one is the one that will make the connection between the optimizer and the parameters of our
neural network.
That is the weights of the neurons of our brain.
And to do this we take our brain which we called CNN.
That's the object we created for our brain.
And so CNN that remember parameters that we go and some parenthesis so that makes the connection between
the optimizer and the weights of the neurons in the brain of our AI.
And then the second argument is a learning rate and that's given by our.
And so here we have to take a smaller raise because we don't want to converge too fast and we want to
have some acceleration and therefore good learning ways that we can take.
Here is a small one that is on point or one that is open 1 percent.
I think that's the same we use for the self-driving car.
All right so now we have a last function atomizer so we are almost ready to start the fall.
Well actually we will start to it right now but just before we do it we're going to decide the size
of the number of epochs we will be training the.
And therefore in creating a new variable here which will respond to this number of abox and let's say
it equal to 100.
That will be way enough to train the AI.
And even that that's the way I will manage to reach the best way before 100.
Like 20 or 30.
Let's see.
But for now let's take 100.
And if we need it we will increase it.
But I don't think that will be necessary.
OK so now that we have our number of airbags we can start to make the follow up.
You know this main for loop of the training when we train over the Xbox.
So for then it's a really viable is going to be book.
That's what we choose for it back in.
Now of course we're going to use the range function to say that we want to go from the first back one
to number of books
plus one because remember the upper bound of a range is not to.
Therefore we want to go up to 100.
Well we have to specify and be the plus one to go up to 100.
All right so CULLEN And now let's get into the loop.
All right so the first thing we're going to do is to do 200 runs of 10 steps.
So each epoch will be 200 runs one after the other of 10 steps and to do this we have this one step
function from our experience replay class and therefore to use this function which is actually a method
because we will get it from our memory object which is an object from the replay memory class to generate
these two hundred runs of 10 steps.
Well we have to say our memory object that I remind we created right here memory is an object to replay
memory class with steps 10 steps and a capacity of 10000.
We created this object and from this object we take.
Well this run step function run steps and we specify two hundred successive runs of 10 steps.
So that will just add each epoch.
Basically one hundred steps.
Now now that we have these 200 steps running at each epoch.
Well it's time to sample some batches from these runs and to sample these batches.
We have another function from our memory which is sample batch and that will exactly generate some batches
from these two hundred runs.
But remember these batches are this time batches of series of transitions.
There is a series of 10 steps as opposed to before where the batches were just some batches of single
transitions here this time.
There are going to be batches of ten steps 10 transitions and therefore Now it's time to get from our
memory.
These random batches.
And to get them we used the simple batch function to which we have to apply the batch size for the batch
size where we can take 32 or even 64 or even 128.
Remember for that says this It's a common practice to use 32.
That's what you will see in general in the neural networks architectures when doing some Bachche learning.
But this time it's quite different.
We were just sending some batches of ten steps.
So it's better to take benches with larger sizes.
So that's why we can take 64 or 128.
So we're going to take 128 and actually this is going to be inside a Fulop because we want to take several
batches and we're taking them in what is returned by the sample batch function.
So this Fulop for that in memory sample batch 128 means that every 128 steps Well our memory will give
us a batch size 128 which will contain actually the last 128 steps that were just run.
We're just getting some batches of size 128 and the learning is going to happen on these batches.
And besides inside these batches we will have eligibility trays running you know to learn every ten
steps.
All right so now inside this loop which is still happening in one epoch.
But now this time we are in a specific batch and so now the first thing we're going to do is we're going
to get our inputs and our targets separately.
And that as I told you is you can do it with one of the tools we implemented which is eligibility tricks
as you can see here.
This illegibly trace function takes a batch as input.
Now we have to batch and returns as output the inputs and the targets.
So right now what we can simply do is create two new variables which are going to be the inputs and
the targets and do these inputs cannot tell us equals exactly what returns this LGBTI trace function
applied to batch to apply this function to the batch of our loop.
And so what we'll do is just eligibility trace applied to the batch of our up.
All right.
So that gets us the input and the toilets but in PI torch there is always something more we have to
do.
And of course this is to convert the inputs of the neural network and also the target into some of variables.
But nor is there is nothing new.
We know how to do it.
We can do it this way we take our inputs then our targets.
And while they will be equal to variable inputs that's for the inputs and variable targets and that's
where the targets.
All right so the inputs of the brain are convert into some torche variables and the targets also are
converted into some variables.
So now we can get the inputs into the neural network.
And why do we need to do this.
That's because the next step is to get the predictions we have in which we have to target.
Now of course we need our predictions because then what happens is that we will compute the loss between
the predictions and the targets.
Let's get these predictions to get then.
Well again this is so simple now we just need to take our brain which is CNN our constitutional news
network and apply it to our inputs.
There we go.
The inputs go into the neural network and the neural network will output the predictions.
Perfect.
So now we have the protections we have to target so we can get the laws and that's the next step.
We're going to introduce a new variable because right now we're going to get the last error which is
different than the last function because we use the laws function to get the last error.
So last error here and that we will get it with the last function applied to our predictions and the
targets that we go see how everything is smooth.
Now everything is logical.
We get input first the targets then thanks to the inputs we get the predictions and then thanks to the
predictions and the toilets we get the last error.
So very logical and smooth.
And now what is the next step.
Well same logical path now that we have the last we can back propagate this last error back into the
neural network to do weights and we do that with the gas grill in the center and to perform to get degrees
in the sense we need our optimizer.
But we already got it here.
Our Adam atomizer.
But now at this point remember what we had to do.
We had to initialize it and to initialize it.
Remember we take our optimizer object.
And then we apply the zero grette method.
There we go we don't forget the parenthesis that's initialized it.
And now next step is to back propagate the last error back into the new network and to do this well
we take our last error and we apply on it the backward method.
So that's exactly to ply backward propagation.
And then finally now that the last error is back propagated into the new one that work well we can update
the weights.
We're still getting great in the sense and to do this remember we take our optimizer and then we apply
the step method.
There we go.
The weights are now dated.
As I told you not only we already did it but now it seems so simple and so natural.
So now we're going to do something fun.
We are going to print the average we work every epoch.
So you know we can keep track of how the AI is going how the training is going.
We want to see the average reward increasing over the steps of the epochs and at first of course there
is this exploitation phase so the average would not increase at the beginning but then once it reaches
the exploitation phase then we'll see the average reward definitely increase and it will increase up
to a certain level which is when it reaches the vest as fast as possible.
So let's start with the premise.
You know we are doing this in one way but we have to go back to the loop here print and then we're going
to print.
Well first but then percent s because we're going to convert everything to a string that's better.
And then we're going to add the average we were and then we add percent S as well then we're going to
close the quote and then we add a percent.
And on the other side you know we can put the variables that are going to be this first person as that
is the epoch here.
And the second variable correspond to the average word which will compute right now.
So the average word viable doesn't exist yet.
We're going to create it right now.
So we are going to use Esti our epoch even if is the number to convert that into his train gets better
and we are going to add s t r that's going to be the average reward.
And so we're going to create a variable that we're going to call a Viji reward.
And now we're going to create this variable and compute it.
OK so let's do this.
That's the only thing we had to do left so EPOC we already have.
Now let's compute average word and we need to compute that right here.
Still in the loop but our the bachelor because now we have our patch samples and we have a training
happening in the batch.
But now that for propagation Plus the backward propagation is done in the batch.
So we are getting back into the epoche loop and we can now compute the cumulative words which we can
do with our steps object because our insteps object contains this function we what steps that allows
us to get the cumulative rewards happening in the steps you know and steps run.
So we are going to use it right now to update the new rewards of the steps and then we will update the
moving average object by adding the cumulative words to the moving every object.
And then we computing the average and that's how we're going to get the average of what we think is
the first thing we need is the word Stribild.
So let's call them the words steps and then as we said we take our end steps object which was I remind
created here and object of the step progress class from our experience replay for instance objects.
Then we add words steps and then some parenthesis.
All right.
That will get us the new cumulative rewards of the steps.
All right but then we need to add these new cumulative rewards in our moving average object and to do
this we have a method this time in the moving average class which is this method that's very simple
we take our moving average object which we created here with 100 steps then we're going to use our add
method.
And then in the Admetus we put our words steps and this will add the rewards of the steps into the moving
average.
All right and finally we can compute the average word and that is well you know that's the variable
here.
So that's what is going to be equal to the average word and to get it we just need to use the average
method this time from our moving average object and that is we do and say that average to say that because
our moving average was already updated with the new word steps that we just added things to the Add
method great.
Now we have our average word so that will populate here.
And this is going to be printed every hour.
So this is going to be very exciting because each episode we're going to keep track of how the air is
going.
You know we're going to see if the average word is increasing.
And speaking of something exciting.
What's very exciting is that I know that it reaches the vest for sure when it reaches 1500.
So what we're going to do now is that if the average reward reaches 1500.
Now if it's getting higher than 1500 Well we're going to print something new and that's going to be
the final print.
The very successful print we're going to print that we can say.
Congratulations to your AI because your AI wins.
So that would be awesome if that happens if we reached non-average word.
Larger than 1500.
We can be 100 percent sure that the AI which is the best.
Maybe it's a little lower than that but I experimented with 1000 and sometimes didn't reach advanced
but with 1500 if we get there well the AI wins for sure.
So we're going to watch that afterwards.
All right so that would be very exciting.
And then that's actually only the end of the curve.
We need to specify that we want to start the training.
If that happens you know if it reaches the vast wealth it no longer needs to train.
So we're going to end all the process with a break if we reach an average world of 1500.
We can finish the whole process and to do this we can use this break here.
All right.
And then one last thing to do when the game finishes that is when the AI wins.
Well we can close environment and so on adding just one last action here which I'm going to call closing
the doom environment.
So to do this we take our Dume environment which we called Doom and remember.
And then we can just add that close and that's all that we'll just close environment which is better
to do once you've finished the game.
All right so we're done.
So I'm so excited to see the results and actually I'm going to have a surprise for you in the next tutorial
while watching the results.
It's going to be pretty exciting.
And so now I guess it's time to play with it and have fun.
So let's hope we can reach this vest as fast as possible.
That's the goal of this new environment.
All right so let's have fun.
Prepare yourself a good coffee or a good tea.
Now stand to sit comfortably in our chair and watch some very cool videos of our planes.
Let's do that in the next tutorial.
And until then enjoy AI.