Hello and welcome to this Python tutorial.
All right.
Now in the next function that we're about implementing We will train the deep neural network that is
inside our artificial intelligence.
So basically we're going to do the whole process for propagation and then back propagation.
So it is we're going to get our output.
We're going to get the target.
We'll compare the output of the target to compute the last error then we're going to back propagate
this last error into the new network.
And using suggested gradient descent who will have to wait according to how much they contributed to
the last error.
So let's do all this for those of you coming from the deep Marine Corps.
This will be good stuff but for the others don't worry I'm going to expand that again.
So we're going to call this new function learn and learn function is going to take several arguments.
First self of course which will refer to the object of the degree in class.
Then we're going to take our batched state for the current state then our batch next state then our
That reward and finally Arbat action.
So why do we take this.
You probably recognized what is this series.
Well that's of course a transition a transition of the market decision process that is at the base.
Did you learn.
And why do we all take them into some batches.
Well that's because you know remember we don't consider the transitions by a series of the top or current
state next state current reward and current action.
We created some simple batches here thanks to the simple function.
And so now our transitions are in the form of first batch for this state.
A second batch for the next date a batch for the reward and a batch for the action.
That's the form of our transition's now and they're all well aligned with respect to time thanks to
this concatenation that we made here with respect to the first dimension.
So the point is now we have these transition of batches one batch for each of the state next days we
watch it in action and we do all this because we're using this experience replay trick.
So that's our deep neural network can learn something.
Remember if you only had a transition's by themselves what it would be some instant learning.
Or if you want some very short memory learning and therefore the mole wouldn't learn anything.
So we have to take these batches from the memory which become our transitions and then eventually we
will get the different outputs for each of the states of the input states and we will do this for the
states and for the next States because we will need both to compute the loss.
I will soon remind the balance equation that is at the heart of the learning algorithm.
So now let's go into the function and let's first get the outputs of the box state.
So I'm going to call this first viable outputs and then we're going to say of course our self-talk not
all so self start.
No because we want to get our model outputs of the input states of the state.
And since our model is actually expecting a batch of input states.
Well we can totally input that state right now for the input of them all.
That's exactly how we initialized the states that are going into the network with torture answer with
this vague dimension for the batch that's perfect.
We now get the outputs of them all.
But then there is another technical trick if we only do sell that model state well we will get the outputs
of all the possible actions you know 0 1 and 2 but that's not what we want.
We are only interested in the actions that were chosen.
The actions that were decided by the network to play at each time and so to get these actually interested
in that is the actions played well we have to use this gather function in which we input one because
we only want the action that was chosen and then we add that action with this one.
And in that section we will gather each time the best action to play for each of the input states of
the state.
We don't want the action that is played the action that is chosen and we get this with this gather one
that action.
But then be careful.
The state here has this fake diamonds and grease on into the batch and that section doesn't have it.
Backstay has it because we used to Unsworth with here but we haven't used any arm squeeze for the actions
so we have to add it here so that the Bache action has the exact same dimension as the state.
So we're going to add a dot and squeeze your right here and actually this is not zero but one because
zero response to Faith is not the state and one will correspond to the examination of the actions.
And finally the last thing we need to do here is we need to kill this fake batch with a squeeze.
Why do we need to do that.
Because now we are out of the neural network.
We have our outputs but we don't want them back.
We want them.
And the simple answer is simple vector.
A vector of output the batches just when we work in the neural network because the neural network is
expecting the format of sensors into a batch.
But now we have our outputs and in the next baluns equation of the deep learning we won't need them
into a batch.
So I'm killing that here and killing the faith dimension to get back the simple form of our outputs.
So I'm just adding here Dot and then squeeze and then since I want to kill the fake lamination corresponding
to the back of the action.
Well since the spacetime engine has index one I'm adding one here.
All right.
And now there we go we have our outputs.
OK.
We have a little warning what is that local variable output is assigned but never used.
That's OK.
We will use it very quickly.
So that's our outputs.
And now we want to get our next outwits So now you might be thinking why do we need the next outputs.
Well to understand this we need to go back to the deep learning algorithm which is right here that is
part of the letak handbook.
So that's the whole diffusion process.
At the beginning we were initializing all the key values and then at each time t.
Well there we go we select the action with soughed Max that's what we did with the select action function.
Then we opened the transition and then as you can see we get the prediction we get the target and we
can be the last.
So why do we need the next output as well.
That's because of the target the target is equal to Ghana times the next output plus the we want and
we will compute the targets right after that since we need the next output of the target.
Let's compute this first.
So again to get the next update very simple the next output is going to be the result of our neural
network when the batched next state is entering it as input.
So very simply we take our model that is our neural network and then this time the input of the neural
network is going to be the batched next state that batched next date.
But now remember if we go back to the early algorithm Well you can see that the next output is the maximum
of the q values for the next state with respect to all the actions.
So right now to get the next output we need to get the maximum of this q values and therefore I'm going
to do here.
Detach you know to detach all the outputs of the model because we have several states in this batch
next States that's the batch of all the next states in all the transitions taken from the random sample
of our memory.
So I'm detaching all of them using the detach function and then I'm taking the max of all these two
values.
And since we're taking the maximum of these two values with respect to the action Well we have to specify
that it is with respect to the action.
And since the action is represented by the index one.
Well again we have to put the next one here and then we have to specify that we're taking the cue values
as T plus 1 that is the next stage and the next state is represented by 0 because the index zero corresponds
to the states and therefore here we need to add brackets with index 0.
That way we get the maximum of the key values of the next states represented by the next zero according
to all the actions that are represented by the index one and now perfect we get our next outputs.
These are a new set.
That's when we had the warning.
But that's fine.
We will use it right now to compute the target.
And speaking of the target that's the next step of this known function.
So there we go.
Target equals.
Now let's get back to our AI AI handbook.
The target is equal to the word plus gamma times the next output is the maximum of the cube values of
the next day.
According to the actions that we can compute that.
So that is equal to self that gamma and self that Gamma was initialized.
Here it is introduce a that's a variable the Virgin Q An object self get to Times the next outputs as
we just said plus Do we want that is the best want.
We're working with batches here so plus batch we want and that's the target.
In one sample of the memory gamma multiplied by the next outputs Plus the reward.
All right.
Perfect.
So now we have our outputs.
We also have our targets and therefore we can compute the loss the loss that is representing the error
of the prediction.
So let's call this last the last two is of course for the temporal difference.
That is again at the heart of Q learning and this tiddy us is going to be equal to the release that
improves much the Cunanan.
That's the last function we will choose for our artificial intelligence.
For those of you coming from the deep green course that's really the last I recommend.
If you want to implement the Coonerty and so how are we going to get this.
Well again we're going to take a function from the functional module represented by F and therefore
here I'm going to use our functional module f ducks and the Hubble us can be obtained thanks to the
function Smoots L-1 loves that one.
So pressing enter.
And that's really the best lost function I recommend for deep learning.
It really improves the Culin.
But this is a function so I'm adding some parenthesis.
And now there is nothing more simple.
The arguments we need to input are the predictions and the targets.
So the predictions of course are outputs because that's the output of the neural network.
No the output of the neural network is what neural network predicts.
So that's the prediction.
So the first argument here is outputs and then the second argument is of course the target.
The thing we're trying to get and it's already computed perfect.
We can directly input target.
Perfect.
Now we have told us just we've got a little tea here.
There we go.
Now the warning should disappear.
Yes.
Perfect.
And now that we have the last error we can back propagators error back into the network to update the
weights with stochastic gradient descent and that's exactly what we're going to do in the next step.
So of course now what we have to do as you might guess is take our optimizing our optimizer which again
we introduce you initialized it.
And that's an atom optimizer which is actually an object of the Atom class and it is already fitted
with the parameters of our model.
And we already chose a learning rate of 0.1 percent to perfect our optimizer is ready but now we need
to play it on the last error to perform stochastic grid in the sense and data weights.
So when working with fighters the first thing we need to do is re-initialize it at each iteration of
the loop.
We must re-initialize the optimizer from one interaction to the other in the loop of this to get the
grid in the set and to re-initialize it.
And each iteration of the loop.
Well we're going to use the following method which is zero.
Here we go.
Zero grad will re-initialize the optimizer at each iteration of the loop.
Then let's not forget the parenthesis.
Perfect.
And now that it is re-initialize Well we can perform backward propagation with our optimizer.
And how do we do that.
Well we take our to the laws and we're going to back propagate it back into the network and to back
propagated into the network.
We need to use the backward function and inside this backward function I recommend to input retain underscore
variables and set it equal to true.
I recommend to do this because this will improve back propagation.
The use of written variables equals true is to free some memory and we need to free the memory because
we are going to go several times on the last so that will definitely improve the training performance.
And finally last step of this learned function is to have data weights according to the back propagation.
That is according to how much the weights contributed to the error.
And to do this we take our optimizer again which was initialized in re-initialize and we use the step
function and simply with this line of code by using the step function.
This will update the weights.
That's this line of code that updates the weights.
This line of code that propagates the error into the neural network and this line of code uses the optimizer
to update the weights and there we go.
We have a learning neural network all right so congratulations.
This was probably the most technical and difficult part of all this dispute or morrow.
I know my torch can be tricky sometimes with ease and squeeze and squeeze but in the end I promise you
will get a very functional neural network and therefore peculiarly model and eventually a great artificial
intelligence.
So now let's move on to the next step or teacher and model which will be the update function that will
obviously update when the AI will discover the new state.
So you know it will discover the new state and then it will receive the reward.
Depending on the action that is displayed and this new state will take care of this with the update
function and will do this in the next tutorial.
Until then enjoy.
I.