Hello and welcome to this tutorial.
All right now we have our AI it is ready to be trained.
And the first step of the training is to set our experience replay.
So we're slowly getting there.
The training and the good news is that we have an implemented version of experience replay.
Besides that is adapted to eligibility trace which I remind is a technique that instead of learning
the q values every transition learns it every 10 transitions.
So basically that's exactly the same as before.
But instead of having a single target a single word for each step we're going to have a cumulative target
of ten steps and a cumulative reward of 10 steps and we will learn on the 10 steps each time.
So we are learning on transitions 10 steps instead of one like before.
And with this I will work wonders and that will make some wonders for the training process.
You know the training will take much less time.
Thanks to this technique but we have to specify inexperience replay that we are learning every 10 steps.
So that's why this experience replay is not a classic implementation of experience replay.
Like do one for the self-driving car.
It is an experience replay implementation taking into account this 10 steps learning and therefore you
will find in this experience replay file two classes one class that makes your AI progress doing ten
steps so that it can sum the rewards are observed on these 10 steps.
That's the first class and we need this class because we need to include these 10 steps in the replay
memory class which is the classroom implement for experience we play and that's how we make sure that
the memory also takes into account the fact that we're learning on 10 steps.
So that's why you will find two classes in this implementation of experience replay but that's only
to take into account that we're learning in 10 steps and that must be taken into account also into memory.
So speaking of our memory let's create it.
We're going to call our memory memory and memory is going to be an object of the replay memory cast
and the replay memory class is a class of this experience replay right now.
And so I'm taking first this felt experience replay conduct and that's where I take the replay memory
class.
Perfect.
And now you can see we have to put two arguments the first argument is and steps which corresponds exactly
to the number of steps on which we're going to learn the key values.
So you know the number of steps on which we accumulate the target and we want.
We're going to have a cumulative target and the cumulative reward and then the second argument is the
capacity that is the size of the memory.
So for example here we can see ten thousands.
So if the capacity is equal to 10000 that means that then we will have a size of 10000 and therefore
that means that we will get a memory of the 10000 steps performed by the eye.
But again we're not going to learn every transition.
We're going to learn every ten steps along these last 10000 steps of the memory and that's exactly this
new feature that we introduce here compared to before.
Before we only had this replay memory trick and here we have this replay memory trick plus this trick
I've learned every ten steps and we're going to learn every ten steps and we're going to do it in the
memory composed of the last 10000 steps.
And this is experienced replay combined to ineligibility traits with 10 steps will considerably improve
the training performance.
So let's end with these two arguments.
The first one is and steps and that will be equal to.
But for now let's say and steps will specify what step is right after that it will actually be an abject
of the other class of this experience replay file which is the end step progress class and that allows
to make progress during ten steps.
And remember during the 10 steps we will sound the words on the ten steps to get the cumulative rewards
over 10 steps.
And that is exactly eligibility test.
So now what we have to do is create this steps here and we create it with the second class that we have
in this experience replay file which is step progress.
So now we're going to create steps this and this will be an object of the step progress class that we
take again from our experience we play.
There we go.
So that's the anti-progress class and now we have to put three arguments as you can see we have to put
the environment which is the environment here that we imported.
Then the second argument is our AI and this will be of course the AI that we built right here in the
U.S. and the last argument is step in this that's where we will specify that we want 10 steps you know
to learn every 10 steps that is every 10 transitions.
So let's help with these arguments.
The first one is the environment and that's doom and all right.
Then the second one is our AI AI and that we counted ai ai.
That's the one here.
So this is just the name of the argument of the step progress class and this ai ai.
Here is our ai ai.
The one that we built and then the last argument is and stack and that is equal to 10.
All right.
So right now we are just taking into account in the memory that there is a learning on 10 steps and
this learning on 10 steps is called eligibility trace.
So we're really working on the advanced stuff here.
But remember that's because we're trying to be Dume that's nothing like making a piece of cake.
So we need these advanced techniques to make it work.
So now we're almost ready.
Before moving on to the next step which will be actually about implementing LGBT trays the only thing
that we have to include is the capacity of course and that is let's say 10000 men we will have a size
of 10000 meaning that the memory will contain the last 10000 steps performed by the AI and that will
allow us to generate some many.
As I remember was a simple function.
You know the memory contains 10000 transitions but to train the eye we're going to sample so many batches
of ten transitions not one compared to before 10 transitions this time and we will sample these mini
batches of 10 transitions into memory composed of the 10000 steps.
All right so now I guess we're ready to move on to the next step which is about implementing eligibility
trace.
So we're going to have some adventure here.
This will not be a simple implementation.
So have a good break and when you're ready we can attack this.
And so then enjoy.