Hello and welcome to the spice Intertoto.
All right so in this new code section we are going to implement experience with play.
So we're going to make a new class which we will call replay memory and that will implement experience
to play exactly as you saw in the intuition lectures.
But first let's give a quick reminder about what is experience replay.
So you know all this artificial intelligence is based on Markov decision processes and Markov decision
processes consist of looking at a series of events.
So events are you know for example going from one state to the next state and tipis one.
But if events were like that well since the next day it is very correlated to the current state.
Well the network would not be doing very well.
So for those coming from the deep learning course that's exactly the same as where we learned our time
series with only one timestep.
It was not learning anything because one timestep was not sufficient enough for a model to learn to
understand long term correlations.
So that's the same here and that's why we have to implement the experience replay.
So how does it work.
Well that's very simple instead of only considering the current state that is only one state at time
t.
We're going to consider more in the past.
So exactly like for lithiums and therefore our series of events will not be as.
And I suppose one this will be for example the one hundred states in the past.
So as T-minus one hundred eighty minus 99 up to as minus 1 and then S-T.
So in other words we put 100 less transitions into what we call the memory and that's why we have a
long term memory as opposed to a short term memory or even should I say an instant memory and that makes
the whole process work much better.
And then once we create this memory of the last 100 events we will simple that it will take some random
batches of these transitions to make our next update.
That is our next move by selecting the next section and therefore in this replay memory class that we're
implementing for experience replay we will make three functions.
First of all the function as usual that's the case for any class.
And so in this function we will define the variables that will be attached to the future instances of
the class that is the future objects that will be created from this class.
And so very simply these variables will be the memory of the 100 transitions to 100 events.
And the capacity that is the 100 number you will be welcome to try a longer memory by increasing the
capacity.
So that's the first function in it function and then we'll make two other functions one push function
to make sure that the memory doesn't ever contain more than 100 transitions.
And for this we'll use the capacity by just doing one simple condition and then eventually we will make
the simple function and that will be of course to sample some transitions in this memory of the last
100 transitions.
All right so let's start by introducing the class.
So as you know we start with class and then we give them to the class we call it replay memory and then
in parenthesis we input object then Cullin and then we go we start with the first function.
The end is function.
So that's exactly the same as before we start with death.
Then two underscores in it two underscores again and then the variables.
So there is of course self which is the variable attached to the future instances of the class of future
objects and then we're going to have another variable for you to be able to try some other experience
with some of the memories and that's going to be the capacity.
So this capacity will simply be the number one hundred because we're going to make experience to play
with the one hundred less transitions.
All right and then Collon And here we go.
Let's go inside the function and let's define the variables of our replay memory object.
So the first one will be self taught capacity and as you probably understood this will be the capacity
that is the maximum number of transitions we want to have in our memory of events.
And this will be equal to the arguments who will input when creating an object of the replay memory
class and therefore that is capacity.
That's the argument of the init function.
So capacity.
So again not to be confused self that capacity is the name of the variable that is attached to the object
and capacity here is the argument who will input when creating an object of the replay memory class.
All right.
And then we have a second voivode.
That's of course the memory.
So self taught Nemec.
All right.
And so what will this memory Voivode be to.
Well this memory is supposed to contain the last 100 events and therefore this should be a simple test.
You know a list which will contain the last 100 events the last 100 transitions and to initialize the
list.
There is nothing more simple we just add some brackets like that.
And here we go.
Our memory is initialized so of course at the beginning of the experiment or more precisely the beginning
of the exploration the memory will be an empty list and then we will put the transitions.
Each time we reach a future state.
And speaking of that that's exactly what we will do with the next function that we're going to call
the push function.
We will make this push function to plant the events in this memory list and then we'll use the capacity
to make sure that this memory list always contains 100 events and never more.
All right so let's do this in the next as well.
And until then I.