Hello and welcome to the store.
I'm super excited because we're about to make the A-3 see brain.
That is the brain of our AI.
And speaking about brains I would like to highlight something.
Remember in the first module we made a simple brain with only fully connected layers.
Then in the second module for Dhoom we made a brain that not only had fully connected layers but also
eyes because we added the convolutional layers which gave eyes to the eye because it could observe the
images and understand what's going on inside.
And now we're going to say that even at a high level because we're going to make a brain that not only
will have eyes and fully connect layers but also memory.
Because as I said in the previous tutorial we're going to add a record on that work inside this big
brain and that will give a long memory to our brain so that it can understand the temporal relationships
the temporal properties of the input images.
So there we go and even more powerful brain I can tell you that the Dymo we're about Intiman right now
is really really powerful and we can see how building AI's and doing deeper and deeper and Forsman learning
is all about getting closer and closer to how human brain works.
You know we started with the basic relationships of the brain with the linear connections.
Then we added eyes and we added the memory.
Who knows what we're going to add in the future models.
You know in 2018 maybe they will add something that will make the brain look like even more like a human
brain.
But already with fully connected lasers eyes and memory we have already a really good and functional
brain.
So let's do it.
Let's make this brain.
So as usual we're going to make a class for that because it's going to have a lot of properties with
the convolutions and science.
So we're going to make it function to initialize all this create all these connections.
And then of course we'll have the Ford function that will of course propagate the signal inside the
brain so that we can get eventually the output.
All right.
Are you ready.
Let's do this.
So class we introduce in class which we call actor credit because of course I'm talking about brains
here.
But let's not forget that we're making with the model which is based on the active critic principle
with separately the actor and the critic so will make actually one of the near-full connections for
the actor and one meaningful connection for the critic.
You'll see how well we'll do that will be actually quite simple.
So active critic and this actor class is going to inherit from the end and that Maggio so that we can
use all the Pitre which tools.
So let's do this to inherit from the end in that module.
Well we need to take first the torch library then that then and then and that and Mudgal.
All right.
That way we inherit from it.
All right.
So there we go with our first function which will be of course in its function.
So we start with in its in its double underscore then this function is going to take as argument self
with the object then the input shape that is the dominations of our input images and we call it non
inputs and the actions phase which is basically the space that contains all the actions.
We also know from this action space we can get a number of actions that is a number of possible actions
which will actually get very soon.
So that's why we also needed that for the arguments.
That's all we need.
And then let's go inside the function and let's create all the variables proper to our brain.
But before we do that remember what we have to do to activate in some way the inheritance that we can
use all the tools from the end of Maggio we have to use the superfunction this way inside of which we
input.
Actor critic that is our class and then come up self with the object All right then and there we go
again with the function.
There we go.
That gives us all the tools that we all need from torch to build our brain.
All right then.
Well it's time to make the eyes of the eye.
That is the convolutions.
So we're going to do it very quickly because we already explained this in details for doom.
Because I remember the day I had ice.
So it's exactly the same we're going to make some conclusions and we will use a very simple architecture
with 32 feature detectors of size three by three stride of two and a padding of one.
So that's a pretty classic architecture but that will actually be enough to you know make sure that
I understand what's going on in the break.
OK.
All right so let's make those convolutions.
So we start with self because the convolutions will be variables of the objects so self that can we
can call it come and there's going to be four convolution so and can this one can one.
And there we go.
We take the N and Maggio dot and then we take the cone to the class because actually convenant will
be an object of disgust.
And then inside first we put the input shape of the images.
So that exactly what we have here so we can copy this and enter it as the first input.
Then the second argument is the number of feature detectors are also the number of kernels.
So we're going to take 32 as we just said classic choice then we need to choose the size of the kernel
that is the number of cells that will slide over the input image.
And so remember we can either take three four or five that's come in choices and we're going to choose
three and then we're going to choose.
Tried to.
And a padding of one that we go that's for the first convolution that goes from the input image to the
first convolutional layer is composed of 32 convoluted images.
So now we are ready to make the second convolution.
So it's actually going to be almost the same so I'm copying this line and basing that below pasted below
again and pasting it one last time because we're going to have four convolutions with almost nothing
to change so we can already replace Here comes one by comes to one by three and comes one by come four.
That will be our four convolutions.
And now of course we need to change some things here but not much because we're going to keep track
of two for each and a pattern of one.
They will all have 32 feature detectors that has 32 outputs convoluted images.
But then here remember this corresponds to the left part of the convolution.
So actually that corresponds to what was at the right part of the previous conclusion you know it's
like a dominoes.
It's really easy.
And therefore here we have to put 32 and here's where we're going to see very easily 32 and 32.
All right so to sum up we start with our input images that has none.
Simon Jones was the first convolution we get 32 convoluted images each one detecting a specific feature
then from these 32 convoluted images we apply the second convolution to get 32 new convoluted images
then same from these 32 new convoluted images.
We apply the third convolution to get 32 new convoluted images again and then eventually from the three
to convoluted images we apply the fourth convolution to get features.
All right.
And this will be enough with this or I will have a supervision it will take the ball very well.
All right so that's it for the convolution So that's it for the eyes.
And now let's take care of the memory.
This new feature of this brain we're implementing as opposed before with to not only it will have a
supervision but also it will have a super memory long memory because we going to see Long short term
memory which is this kind of record or a neural network that gives to your model some kind of a long
memory so that it can learn some long temporal relationships from the past so saying we're going to
create new variables from time with self and this new variable we're going to call it simply because
this will correspond to the LACMA network inside the brain.
So SVM and out before we write the code for the LCN let's make sure we understand what this LACMA part
of the brain will do.
So as we understood this LCN is used to learn the temporal properties of the input input images.
So for example if the ball hits a the LACMA will encode the balance.
So that's the first thing to understand.
It will kind of encode what's happening in the game.
Then the next important thing to understand when we implemented that ISTM is that we need to choose
in-order of the temporal dependencies and here since we're going to feed our neural network with a sequence
of four images then that means that we can already learn some temporal dependencies of order for that
is some temporal dependencies where what happens at 1:20 depends on what happens at time T.
T minus 1 T minus 2 and T minus 3 so that we can definitely do that.
But the good news is that we're going to use analysis YEM and therefore we will be able to learn some
even more complex temporal relationships.
For example we can learn some simple properties where what happens at 1:20 will depend on what happens
at time T.
T minus 20 minutes to do minus three down to T minus.
And that's the long part in the long short term memory with this assay and we can learn some very complex
temporal relationships.
All right so let's add our LCN.
So to do this we're going to use the N in module and then we're going to add the class as T.N. cell
which will create this as an object which will represent DST and part of the new network because right
now what's also important to understand is that we're making a C R and then you know convolutional record
a new network and the arland part comes after the CNN part and therefore right now what we need to input
in this LACMA cell is first the size of the output after the convolution.
So that is 32 times three times three.
So this 32 times three times three is actually the output after the four convolutions here.
But that becomes the input of the RNA the LSD and that work.
And now why is the output of the four convolutions half the size three two times three times three.
Well don't worry it's not that direct.
It's actually not a simple formula but there is a formula to compute this number of output neurons.
After flattening the pooled and convoluted images of deconvolution.
But if we gathered the terms of this big formula Well we get 32 times three times three.
I didn't want to spend too much time on this because we have a lot to do more.
And besides we already made a function to compute this number.
Remember it was for doom when we made this count neurons function.
So you can reuse it if you want.
You're not convinced but that is just what we get after gathering the terms of this big formula like
computing the number of outputs.
So that's for the first argument and then the second argument is going to be the number of output neurons
of the.
And we're going to go for 256 OK.
And so what does that mean now that means that now we have a vector that encodes each event of the game
or in other words we have an encoded state and so that is now that we can make the separation between
the actor and the critic because you know we're going to make actually two separate new networks one
for the actor and one for the critic but they will be the same encoding of the images and the temporal
relationships for these two neural networks.
So this is the common part that we do for these two new networks.
This will be the same beginning for the two new networks but now things are going to change for the
actor and the creek because we're going to make one in the full connection for the actor and one different
than near-full connection for the craic.
So let's take a quick break and let's do that in the next little.
Until then enjoy AI.