Hello and welcome to this Python tutorial.
All right we have one last function to implement in our replay memory class.
That's the simple function and that's of course to get some random samples from our memory.
And therefore this function will return these random samples.
All right so let's into it.
We are going to call it simple.
Here we go.
And this function takes two arguments as input.
The first one as usual self our future object to replay memory class and the second argument is can
you try to guess.
Well we're taking some samples of fixed size and therefore we need to choose a size for samples and
more precisely we call it a batch size.
So that's the name we're going to give to our second argument batch size.
And there we go.
We have our two arguments and now we can implement the simple function.
So now I just want to warn you this is going to get a little technical but I'll try my best to explain.
So we're going to start by creating the samples variable.
This is just the variable that will contain the samples of the memory.
All right so simple is equal.
And so now how are we going to get these samples.
Well first of all we have to take our memory because we are getting the samples from our memory.
Then we will probably need the batch size because the samples are going to get contained batch size
Elon's.
So we need memory we need batch size and then we need some by torche or bison tricks to get a good format
of these samples.
So what I'm going to do I'm going to write the line of code and then I'm going to explain that element
by element.
So let's do it.
I'm starting by taking a zip function I'm going to explain very soon what it does.
And inside this function I'm going to add a star.
I'm going to expand that as well.
The Star and random thought Central so random as you might have guessed is the random library that we
imported here.
So that's the main reason why we had to import this random libraries because we're taking some random
samples.
So from this random library we're going to use the simple function.
So this is our variables and this is a function so I'm going to add some parenthesis.
And now as you can see sample is a function and we have to put some arguments so that you can see the
first argument itself.
And actually speaking of self this corresponds to self-talk memory the memory of our future instance
object of our replay memory for us.
So I'm going to add here self that memory and then the second argument is as you might have guessed
the size of the Bechuana take randomly from our memory and that we gave it a name that is batch size.
So the second argument is going to be Bachche sucks.
All right.
So the line of code is typed and I'm going to explain what it does.
So first of all with this random dot simple function we are taking some random samples from the memory
that had a fixed size of that size.
So that's understandable.
But then what does this Zipp star function does.
Well there is no mystery about it.
It is just like reshape function.
So for example I'm going to add a little come here just to explain that I'm going to remove it.
So let's say that for example we have a list of the following elements for example.
First one two three.
And then the second element.
Four five six.
So we have a list of two doubles of three elements one two three four five six.
Well then if I apply the zip function with the star on it.
Well what would become so Zip star list is going to be equal to a new list but of a different shape
and different shape is going to be one for then two three and then five six.
All right.
So that's just what it does.
It just reshapes your list.
OK so now that you understand what this Zipp star list does.
Well now let's explain why we had to do it.
So as you understood we are going to add the events to the memory and the events have the form.
First the state then the action and then the reward.
But for our algorithm we don't want this format we actually want our samples to have the following format.
A format is composed of three samples one sample for the states one sample for the actions and one sample
for the reward.
So for example let's say that this one to three is state one action one.
We want one and then state to action two and we were to well what we want is one batch for each one
batch for state one and state two.
One is a match for action one in action two and a third that for we were one and we were two.
That's just the format that is going to be expected next because then we'll wrap these batches into
a pie towards horrible impact which far we will remember is a variable that contains both a tensor and
a gradient.
And that in order to be able to differentiate with respect to a tensor to be able to differentiate with
respect to intenser within the structure of a horrible containing tensor and a gradient.
Again that's how Pi torch works.
So to summarize we creating one batch for each of the state's actions and rewards and then we're going
to put each of these vetches separately into some bytes which Horrible's which each one will get a gradient
so that eventually we'll be able to differentiate each of them.
All right so that's the purpose of the Zipp function.
So let me just remove this comment.
And now the only thing that we have to do that is to return the samples.
So as I just explained we cannot return the samples directly for the simple reason that we want to put
the samples into a by torche viable.
So to do this for each of the samples we're going to use the map function and this map function will
do the mapping from the samples to torture variables that will contain a tensor and a gradient.
So as you can see this map function takes several arguments.
The first argument is a function and this function is going to be the function that will convert the
samples into some torche variables.
And the second argument is what we want to apply this function onto.
So that will be the arguments of this function and therefore what is it going to be.
That's of course going to be the samples.
So the second argument here is going to be the symbols.
But then let's define the function on which we want to apply.
Each of the symbols.
So to define a function here we need to first give a name to the function which will call lambda.
That's just a name and giving Lenda then X which is going to be the variable of this function.
So that is just a name and giving for the variable and then.
And here we give the expression of the function that is what we want this lambda function to return
and see what it is going to be well it's supposed to be something that will convert our samples into
a torch variable and to do this.
We already mentioned it in some previous tutorials.
Well we have the Voivode function for them.
The Voice will function we'll make that conversion from a torch dancer to a variable that will contain
the sensor and the greatest.
So the first thing I'm going to add here is variable variable inside of which I'm going to convert X
because X is going to be the simples ones that will be applied onto the samples.
But then that's all there is one last technical thing that we need to implement is the fact that for
each batch which is contained in the sample for example the batch of actions a 1 8 2 3 and the other
actions we have to concatenate it with respect to the first time engine which corresponds to the States.
And why do we have to make this concatenation.
It's just for everything to be well aligned.
That is that in each row to state the action and the reward corresponds to the same time t..
So that eventually we get a list of batches all well-aligned and each batch is a pie towards Voivod.
So how can we make this concatenation.
Well we need to use the cat function from the torch library.
So we're going to add here torch to which we add that gets applied to X but then in this function we
need to specify the dimension with respect to which we want to make that concatenation.
And as I just mentioned this is the first time engine that has index era.
And here we go we have our function ready this Lunda that function will take the samples concatenate
them with respect to the first time engine and then eventually we convert the sensors into some torche
variables that contains both a tensor and a gradient so that later when we apply to castigate in the
sense we will be able to differentiate to have data weights.
All right so this function is ready.
And then here is the second argument of the map function.
We need to specify onto what we want to apply this function and that is on all our samples.
There we go.
We will apply this lambda function on all the samples so that eventually we obtain a list of matches
where each match is a PI torch viable.
All right so that was quite technical but now at least everything will work well.
We want to use this technique.
Afterwards we only use it here so if you don't want to have a deep understanding of the technical details
here well that's fine you can just copy these three lines of code to simple your memory if you want
to make an artificial intelligence with by torch it as you want.
But now the good news is that we're done with this replaying memory class experience replay is now implemented
and we can move on to the next and final class which will be the whole security mode.
So in this model we will have of course our network who will experience replay and then all the rest
of the security algorithm.
So it's going to be a much bigger class.
We're going to make about 10 functions but that's only because we're doing this step by step so that
you can understand better what's going on.
So I can't wait to implement our security model.
And until then I.