Hello and welcome to this tutorial now we're going to make this second function to initialize the weights.
And this one will be used to get an optimal learning.
Actually these weights.
So the second function we're going to call it weights underscore in it and it will take as argument
the object and which will represent the neural network.
So that's all.
And then colon and now let's get inside the function to define what we wanted to do.
So basically what we wanted to do is initialize the weights of the new network in such a way that we
get an optimal learning.
So this will not seem particularly intuitive.
This is based on research papers and experiments.
We are going to initialize the weights in a specific way that we haven't seen before but believe me
that will optimize the learning process.
So we just implemented without getting into the details of why we initialize the weights this way.
And so we're going to start by using a trick which will be used later to make the distinction between
the convolution and the full connection because you know remember our AI will have eyes and therefore
it will have some convolutional layers.
And of course it will also have some fully connected layers and we will have a different initialization
of the weights for these two types of connections.
So we're going to use this trick to separate these two kinds of connections and then we'll use some
conditions to get the different initialization for each of these connections.
So this trick is to create a new variable that we're going to call last name and that will be called
to an object so and represents the neural network.
But it's an object we'll see that later.
And we're going to get the special attribute from this object which will be Kolesnik with double underscore
first class double underscore dot double underscore again name and almost there.
Another double underscore.
So that's a pretty ugly trick to look for the type of connection of our new network object but that
will give us exactly what you want.
You're going to see it's going to make sense when we start if conditions.
And by the way speaking of conditions we can stop them right now.
And so what we're going to do now is start the first condition which will get us the first case.
That is if the connection is a convolution and so to write this condition right if class name that's
fine.
Here we use a method the find method find and inside when put in quotes for convolution.
And so if class name does find can is we're going to do different than minus one that is actually if
we have a convolution because Manasquan means no.
Well in that case we will do a special initialization of the weights.
So this condition here means if we have a convolution connection.
So in that case what we do is run this specific initialization of the way we want to do.
And so that's where all the not intuitive things will come.
We will start by creating a variable that we're going to call it.
And this court shape so weight underscores shape will be a list that will basically contains the shape
of the weights in our new network.
And so we to use the list function to create a list.
And inside we're going to put in the neural network that weight which will be the weight of the neural
network.
But in the convolution connection and to get the shape of these weights we use another attribute which
is that data and then size size will get us the shape of these weights in the convolution connection.
So now weight shape contains in a list the shape of the weights and the convolution connections of our
network and.
All right so we have weight shape then to initialize the weights of this convolution connection.
We're going to need two values.
First is the product of the first dimension by the second nomination by this third dimension.
So that's what we're going to get right now and then we'll also need to get the zeros time ancient times
the second time and sometimes the third dimension and then we'll use these two values in the competition
of how we initialize the weights.
So let's get this through this first product.
We call it fun in and that will be equal to the product and we going to use the prod function which
is a function by non-Thai which has a shortcut and P So MP that PRUD and inside prod we input what we
want to make the product.
And so as we said that is the diamond and one two and three of our weight shape.
And so to get this we can take a wave shape and get the indexes of these three line engines.
And so we set it Simonton one up to 10 inch and three included.
So a dungeon for excluded.
And that's how we can get it for the upper bound here is not included.
So that is what we want then Same for fun out as we said Fan out it's going to be the product of the
damage in zero time dimension two times that I mentioned three.
And so here we can get indexed from two included to four excluded.
So that will get as the product of time and two and three and then we can multiply it by design and
zero which we can access with whaleship zero of index zero.
So to sum up this is the one time two times in three and just below we have zero times two times in
three of our weight check list of weights.
All right so now we're going to use these two values fan in and fan out to proceed to the initialization
because we're going to compute a new value as we're going to call W bound and that's equal to the square
root which we can get with a function and P from and by that as q r t.
Second like before.
So the square root of 6 divided by fanning out.
So fan in let's fan out that we got this W down here represents in some way the size of the tens of
weights.
And why did we get this.
It's because then what we were just about to do now is we want to generate some random weights that
are inversely proportional to the size of the tensor of weights.
Because indeed what we're about to do now is take our new network and then get its weight.
So by still taking the attribute weight then access its data that is the tensor itself.
And then from this tensor of weights we're going to generate some random weights that are inversely
proportional to the size of the tensor weights.
And so in this uniform function now we have to put a lower bound which will be minus W bound and the
upper bound which will be plus W back.
OK so that's for the weights.
And now we need to initialize the bias and good news for the bias.
It's going to be much more simple.
We are going to initialize them all with zeros so to get these buys we take them from our model of course
that is our new network.
And then the attribute for the bias is bias then same with access to the data.
And then we're going to use a method which is the Phil underscore method which as you might have guessed
is used to fill the tensor of bias with zeros well with the rules we have to specify that we want to
fill it with zeroes here.
And so that's why I'm putting here Zero.
All right so to summarize we generate some random weights inversely proportional to the size of the
tensor weights and we initialized device with zeros.
All right so that was for the Initialize action at the convolution connections.
Now we need to do the same for the full connection.
And so we're going to add new condition and if same we take this trick we use first class name that
is this variable that contains the different names of the connections.
So if class name that same we use the find method to which when put in quotes this time a full connection
that is a classic linear connection in a classic artificial neural network.
And so the name for that is linear and saying we're going to use this trick to say that we want it to
be different than minus one so that's this end of class thing fine linen are different than minus one
means if the connection is in the there that is if we have a classical connection.
So in that case how are we going to initialize the weights.
Well it's going to be quite the same.
We're going to introduce whaleship Voivode which will not erase the first one because we will either
in this case or that case so it will not be the same so we can totally reverse that then same we're
going to introduce a fan in variable which this time will not be equal to the product of these three
dimensions.
But actually this time it will be equal to simply design mention one.
And that's because for the full connection there is less connections than in a convolution connection.
Now is this an intuition lectures in the end and in the CNN section there is less mention for a full
connection than for a convolution.
So basically we just take this time in one year then say we're going to have a fan out variable which
will then use to compute W. bounds and this fan out domination is going to be weights shape of index
0.
That is the diamonds zero.
All right then to GWB and it's going to be the same.
It's going to be the square root of 6 divided by the sum offending and find out.
So there we go.
And then the good news is that it's exactly the same as previously we use the uniform function for the
weights and the fill function for the bias to get the same kind of initialization this time with a different
fan in and fan out and therefore different w that.
So that's the same principle that's the same idea.
The only thing that changes here is that we have less dominations for the full connection and therefore
more simple consideration of this bound of the weights here to generate these random weights.
But the good news is that now it's really not only these weights and it functions very.
But now we have two tools.
And so we are ready to start building the brain.
So I can't wait.
This will be of course the most exciting part.
This was just to warm up and get us ready for the big thing.
So we'll take care of that in the next tutorial.
Well actually it's going to take several tutorials of course.
We'll start by making the eyes.
And then remember we will add an illustration to learn the temporal properties of the input and then
we'll take care of the actor and the critic.
And that's where we'll use this to function normalized comes initialiser and weights in it.
So I can't wait to do that.
We're going to make something very powerful now.
So get ready for it.
Until then enjoy AI.