Hello and welcome to the story.
All right so after we made these four convolutions and the LCN we now have an encoded state that is
going to be the input of these two neural networks that we're going to make for the actor and the critic.
And speaking of them the only thing that we have to do now is just create tools in our full connections.
One for the actor and one for the critic.
But before we do that we need to get the number of possible actions.
And so I'm going to call you variable here that is not going to be a variable of the object.
So I'm not going to use self here but I'm going to create viable non outputs which will represent a
number of possible actions and to get it.
Well we can get it from the action space.
So we take our action space which will be the input of the function when we create the object.
And then we add that and to get this number of possible actions.
And now the actor and the critic will take separately the same input that is the output of this whole
process here with the convolutions and GLSEN.
So it will take the same input which is an encoded state but then they will have two different linear
connections so that we get eventually actually two neural networks one for the two and one for a critic.
So let's make these two separate neural networks.
But since we already did the big job with the encoding here well what we simply need to do is just create
two objects one in the full connection for the actor and one other linear connection for the critic.
And so that's exactly what I'm going to do.
I'm going to create two objects now a first object for the linear connection of the critic which I'm
going to call critic and the score Linnie are and to create this linear connection.
You know how to do it we simply need to take the engine module and then the linear class to which we
have two inputs.
Well the input neurons which are the outputs of all this including here with the convolutions and the
GM that is 256 neurons.
So when I put two hundred and fifty six here and then we're going to have one output because remember
the output of the neural network for the critic is the value of the function applied to the input state
to the inputs encoded states that we made here.
So if we call the input state s that is the output of all this well the output of the neural network
of the critic will be VS and therefore it has one dimension.
It's just a value.
And so here we put one.
And remember this is what is shared among the actors so that they can get some common information that
they can use to play their action in a more relevant way.
OK so that's for the neural network of the critic.
And now let's make the new network of the actor and therefore I'm in here self taught actor linear and
same we already have the inputs encoded States.
So now we simply need to add a linear connection and therefore saying we take the in a module then the
linear class and now saying this new network of the actor will take the encoded state that has the size
of 256 so 256 here.
But then the output is going to be different because of course you know that the output of the neural
network for the actor other key involves the key values of input states.
The one that we could hear and the action plate.
So again if we call this encoded state that we mean here as an action played a the output of this neural
network actually there will be q as a.
And since you know we have one huge value for each action then therefore we have no outputs.
Q values and therefore the output here is going to be non outputs because no output is actually the
number of humans.
Okay perfect.
So if you want I can write for you output here.
The critic is the s well as is the encoded state and for the actor the output is cute as all right.
So that's very important to understand this distinction here and to understand that we therefore have
two separate news networks.
One for the critic and one for the actor Okay perfect.
So we are almost done with this function.
Now the most important thing is done.
The only remaining thing that we have to do is to initialize all the weights of those two neural networks
and all the bias.
And of course to do that we're going to use the two functions that we made earlier that is normalized
columns initialiser and the weights in it.
So let's do that quickly.
It's going to be pretty easy and pretty fast.
So first we're going to initialize some random weights and to do this we're going to apply the weights
in it function to our object.
So here we have to take self to get our object and to object.
We apply the weight in its function.
Therefore inside we just need to put the weights in that function and then we get that will apply this
function to our object and by doing this we are just initializing some random weights to get a future
optimal learning of these weights.
And now what we have to do is make a special normalization for the actor and the critic.
But remember I think I told this in the previous tutorials we're not going to set a same variance for
the X in acrylic yakked you will get a small standard deviation small variance.
And the critics will get a big one.
And why do we do this.
What's the purpose of giving a small standard deviation of the weights for the actor and the large standard
deviation of the way for the critic.
Well that allows to manage to deal exploration vs exploitation.
That's exactly the purpose of doing this by giving a small variance to the actor in a larger audience
to critique.
We will have a good management of exploration vs exploitation.
So let's do this let's first take care of the actor.
So we take self or object then we're going to take the neural network of our actor which is actor Linnea
are then we are going to access the weights of this new network of the actor and remember to access
the data of the weights we need that data.
All right.
So with this we get the weights and now we're going to use our function normalized comb's initialiser.
So I'm copying this pasting here and we're going to enter an argument.
The standard deviation we want these weights to have.
But first remember this function takes two arguments.
First the way we want to initialize.
So we simply need to take that again and base that here.
And the second argument is the standard deviation.
We want these weights to have.
So as we said we want a small standard deviation to the actor in a small one it's going to be 0.01 perfect.
So that's where the weights of the neural network of the actor.
Now let's take care of the bias of the new that work of the actor.
And therefore here we're going to do almost the same thing we're going to copy this paste that below.
Replace weight by buyers to access the buyers and after data we're simply going to add fill and remember
inside when put zero because we want all devices to be initialized with zero.
So actually I don't think this line is necessary because as you remember the buyers are already initialized
to zero with this fill function in the wait function.
So you know we're doing this just to make sure that the buyers are actually initialized to zero.
But I think it's already done here.
But anyway now we are 100 percent sure.
All right now we're going to do the same for the critic.
So let's be efficient and let's cover these two lines.
Let's face them here and here we're just going to replace actor by critic.
Same here.
And now the only thing that we have to change is just the standard deviation we want the weights of
the neural network for the critic to have.
And as you remember once this time a large standard deviation instead of open or one we will input one
so there we go we have a small standard deviation for the weights of the new work of the actor and a
large standard deviation for the weights.
And then when we get at the critic.
And of course that's something we get to replace active here by credit.
All right now we get cool.
So now we have two remaining things to do first is to initialize also the bias of the team and to do
this we take our object self because the LACMA belongs to our object and we'd say or as T.N. then that
and then we're going to get the two types of buyers that are in the last.
That's bias and the score by age and the other one is based on the score age age.
That's the two types of bias in the CME and same there are going to be initialized to zero.
So first we access to the data and then we use the fill underscore function to fill all these buyers
with zeroes initialize them with yours.
Right.
And now for the second group of buyers we are here the same get replaced by age by age age.
All right.
That's initialises the bias of the zeros and now the last thing we need to do is to use a method that
is inherited from the end and that module that is the train method.
And basically there is just a method that puts the module in treatment.
So what's the use of it.
Well the use is that it allows to activate if there is any drop out in the bathroom ligations.
And so to use it we just add self thought train and that puts the module in trammeled perfect.
So we're done with the init function.
We have our convolutions we have the CME we have our two separate neural networks for the critic and
the actor and all the weights and bias are well initialized.
So that's all good.
We are ready to move on to the next step which is to make the forward function that will of course forward
propagate the signal from the very beginning with the original input images throughout all the brain
until we get the output.
So let's do that in the next tutorial.
And until then enjoy AI.