Hello and welcome to this pattern it's horror.
All right so we just updated the memory after reaching the new state.
And now let's take care of the next day.
According to you now what is going to be the next update.
Well basically we're done with one transition we have dated the last element of the transition which
is the new state.
So now it's like we're starting all over again.
And when we were starting all over again.
It's like you know we are in this new state of the environment and so what do we need to do now naturally.
Well of course it's to play an action because we already got the observation of the news States.
Now the thing that we have to do is play an action and therefore what we need to do now is of course
use the select action function to play the action.
So let's do it let's create a new Voivode action and let's play the action with the select action so
I'm taking.
Well first self to specify that the select action function is a method of the object of the class that
will be created.
So a self that selects action.
Here we go.
So that's the next action.
And then of course since the select action function takes the state as input because of course the select
action function will return the output of the neural network when the current input state entered the
neural network.
So we have to input the input stage here and since that's the states that we just reached in the environment
right now where the input state is of course you state because this state that we just reached at the
time we right now is Newstead.
So in this select action function I mean putting new sticks.
All right so with this line of code we simply play the new action after reaching the new state.
OK and now that we played an action Well we get the reward and therefore we get feedback with the reward.
And therefore if we have more than 100 elements in the memory Well it would be time to learn.
And therefore what we must do now is what logically comes after selecting an action which is of course
to lower the AI needs to start learning.
If it is doing the things the right way.
And now since it just played the action well we're going to make the AI learn from its actions in the
last 100 events.
But before we apply this learned function we have to make this condition to make sure that we already
have reached more than 100 events because you know we're learning from the random samples of the memory.
You know we have this huge memory of 10000 elements.
We're taking some random samples of the memory of 100 elements and the AI is learning from the information
contained in this sample of 100 random transitions.
So let's just make this if condition to make sure that the number of elements of the memory of that
memory and then be careful just a little trick here self-taught memory is the object of your replay
memory class but then the replay memory class has an attribute which is memory.
So in fact we need to take some of the memory that memory the first memory is the object of the replay
memory class and the second memory is the attribute here self that memory.
So if the number of elements in the memory is well we want it to be larger than 100 then Cullin And
then what happens.
Well we can learn but for learning we need to get this random sample of 100 transitions and this we
can get with the simple function.
And since the simple function returns the different batches to states at time t this data 20 plus one
the actions of time t and we was at 20.
Well what we need to do now is create some new revivals which are going to be the batch of the states
at time t the batch of the next dates the batch of the words and the batch of the actions and we can
simply give the same names as we gave for the arguments here and they are here.
And these variables will be equal to what the simple function returns because it returns exactly these
batches and the state's next takes words and actions.
So what we simply need to do now is get first our memory object and then from this memory object we
are going to use the simple method which will take as inputs.
Well the number of transitions we want our AI to learn from that is 100.
That's why we made sure that the we had more than one hundred transitions.
So it's going to learn from 100 transitions of the memory.
So the learning will be much better.
And so now let's make this really happen.
Well since the learn method is a method of our in class.
Well we need to access this Learn method from the future objects that will be created from a different
class and therefore what we need to take is self.
Self refers to that objective to do during class and then learn as this learned method learned method
to which when put Of course these guys here the bad state the Belgian state the natural world and the
Bachche action.
These are our batches sampled from our memory and we get 100 of them because we have 100 transitions
from these 100 transitions we take 100 States 100 next states 100 reward and 100 actions let's face
the here and there we go now the learning will happen.
It will happen from all these random batches.
Perfect.
And now what we need to do is the very last updates after you know reaching a new state and playing
in action.
Well we got you actually to play but we still didn't have that reaction.
That is our self that last action voivode.
So let's make sure we don't forget this.
Let's do it right now.
We will update the last action self that last action equals and of course action the action that we
just stay here with this select action function.
All right now the last section is updated then.
Same for the new state.
We reached the new state but we haven't updated the last date yet because of course the last was before
the state at time t.
But since now we reached the new state surplus when it's time to pass one.
Well the last it becomes this you said here.
Therefore we need updated as well.
Self that last state equals our new state.
There we go.
And now what do we need to date.
Well there is only one thing left.
That's of course the word and the word is exactly the word we get in reality.
So that will be the argument of this function which if we made the connection to our map will be the
last word.
That is the word we get right after playing the action in this reached new states.
So if we go on to some sound this last word will be that minus one if we get further from the go we
will get a slightly bad word minus 0.2 if we get closer to the goal.
We will get a slightly good reward 0.1.
And if we get too close to an edge of the map well that will be about punishment.
We will get minus one for each.
So that's the last word we get.
In reality that is when that happens for real on the map.
And this will be the argument of the function.
The last word here.
That's exactly this one.
And so since this is the argument of the update function well that corresponds to this we weren't here
and therefore our self that last word variable initialized at the beginning in this function becomes
the new word we get in reality that is the word or that's the same last word.
All right.
Now we updated our last word.
And now since we just got our last word.
Well we can now date the world.
You remember the war when they we initialized here as one of the variable of the object of our class.
That's the window that's going to keep track of how this train is going by taking the average of the
last 100 reward.
So you know it will be like a sliding window showing us how the mean of the world is evolving.
And so since we just got our last word.
Well we can update the we were window into how do we update it.
Well we simply need to append this last word to the window and therefore what I'm going to do is take
my war window self that we word when they hear this and I'm going to use the append function.
And inside the open function we need to input the element we want to append to the we were when doing
that of course do we want.
All right perfect.
And then since this war window is going to have a fixed size you know it's not going to be a growing
window it's going to be a window of fixed size sliding with time to show the evolution of the world.
And so now we need to decide for a size of this winter.
And it's simply the number of means of the reward we will have in this window.
And so for example let's get you know the last 1000 means of the last 100 words.
And so to make sure of it we're going to add if then plan then we take our work window and we simply
add here if the number of elements in the window is larger than 1000.
Well what we want to do is delete the first element of this who our window and the first element of
this were window has to index zero.
All right.
Now we make sure that this war window will never get more than 1000 elements.
There is one thousand means of the last 100 words that's perfect this will be a window of fixed size
so that we can see if the mean of the word is increasing.
And therefore if the training is going well and accordingly the court does what we want.
Perfect.
And now one tiny little thing to do left according to you what is it going to be.
Well remember this update function not only updates the different elements of the transition in the
war window but also it returns the action that was played when reaching this new state.
That's why we have and then that action equals bring that date that we're less signal and therefore
it's supposed to return something and the something that is supposed to return is of course the action.
So the simple last thing we need to do here is just return action the action that was just played when
reaching the new stage.
And that's our update function is ready.
It's going to do all the required updates and it will turn the action when reaching the new stage.
That's perfect.
That was the last difficult action to make for all this a process.
Now the rest will be good stuff.
We will just make a core function to return the means that we want in the window.
Then we will make a safe function to save the brain of the car whenever you want to quit the application
and go back to it.
And of course since you want to be able to load the brain of your car when you get back to it get back
to the application.
Well we will end up by making a load function which will load your model after you save your model with
the same function.
So three functions to do that but it's going to be simple.
And then we'll have the most exciting section of this first module that is the demo we will see if the
air works.
We will see if the car reaches the goals and we will see how we can improve it and then eventually you
will have to build your first AI.
So I can't wait to start the demo.
Let's make this three functions first and until then I.