Hello and welcome to the Statoil.
OK so we just computed the entropy and added it to the entropies list.
And now what we're going to do is take a random drop of an action according to the distribution of probabilities
of the next.
So let's do this.
That's the next step.
We are still in the loop because we're still running on the steps here.
And so you now know how to play the action.
We will first introduce a variable for the action called action and then we take our distribution of
probabilities and we're going to use the multi no neural function to take a random draw from this distribution
of probabilities and then we add that data.
So it's important to note that the action will actually be a tensor with only one value but you should
not see this as a simple value.
You should see this as a tensor damnations one by one that contains this value for the action.
And that's because it isn't squeezed out still in the same for loop.
We're going to get the log probability associated to the action that was just played.
And so when I'm dating my luck probability here by taking the previous one the previous luck from that
we computed here and then I'm going to use the other method to which I'm going to input 1 and the action
that was just playing because we want to get the luck probability that is associated to this action.
And so the second argument here I'm going to put my action but there has to be as a torture horrible
as required by the gathered function and the gathered function just indexes with a tensor integer.
All right so now we just got the look of associated to the action that was displayed.
And now the next step is to append what we got to the list here.
So we got the value.
That's what we got here as the output of the model.
Then we also got the lock problem.
So we are going to add the lock to the lock props list.
We already append the entropy to the entropy is less good and the rewards will get it afterwards.
So we will now open to value and the look up to the values list and the law process.
Let's do this.
We take our values list we add that we use the spend function and we add the value that was returned
by the model perfect then Same for the lock probs We just got our new props and we are going to append
it to the lock props list.
And so in this append function we can put a log from our luck probably it was just computed here.
All right so our lists are now well updated.
Now what we're going to do is play the action because actually right here we selected the action by
taking a random draw from the distribution of probabilities here.
But we actually haven't played it yet and we're going to play it now so that we can reach the new state
and therefore get the new transition and to play it.
We're going to take our environment because we play the action in our environment then we're going to
use the step method.
And inside we specify the action that was selected to play it and to do this we take our action and
we add that none by because that's what is expected that is the function.
Ok but this returns actually the new state and also the new reward because by reaching the new state
we get a new reward and also we get a new value for Dunn to know if the game is done or not.
All right so with this we play the action we reach a new state and we get a reward and we know if we're
done with the game.
And speaking of being done with the game.
Well we're just going to add something here that will make sure that an agent is not stacked in some
state.
And to do that we're going to update that done very well the following way.
Well it's going to be equal to done or we're going to add a condition saying that the episode of the
game should not last too much time and we will see in the main function that there will be a max length
parameter which will be equal to 10000.
And we don't want an episode to last more than 10000 units.
So we're going to hear episode length which is the length of an episode and we're going to write a condition
larger than max episode Lex that we haven't actually said this at length.
We are getting it from our parameters for an ending here Paramjit but Ramstad.
Max is at length.
So this means that if the game is done or the length of the episode is larger than the maximum length
of episode set which will be equal to 10000.
Well the game will be done and we will start a new game.
OK so that's just a precaution.
And speaking of precaution we're going to add another precaution to clamp the reward between minus 1
and plus 1.
We already got the we were here but we want to make sure that the reward is between minus 1 and plus
1.
And to do this we simply need to update the reward by doing this taking the max then taking the men
of reward and 1.
And here we take the max of the minimum of reward and 1 and minus 1 and that will make sure the reward
is between minus one plus one.
All right.
So another percussion.
And now we just want to check if the game is done in which case we will restart the environment.
And why do we need to check that now it's because we just reached a new state.
We just passed a new transition.
So we need to check that after passing this new transition.
Well the game is not done so if done again if done then in that case we will restart the environments
by setting the episode length to zero.
And also the state will be re-initialize to re-initialize as we take our environment and we use the
reset function OK.
Now we get out of this condition that was just checking.
And now what we're going to do is since we reached a new state while this new state is right now and
then by Ray because remember the states are the input images which originally are named by arrays.
And so now what we have to do is to convert the new state into a tortured answer.
So we're going to update our state and we're going to use the torch library.
And of course the from non-Thai function to convert this non-payers state the input images into a torch
sensor.
Perfect.
And now the last thing we need to do before getting out of this for loop that is the loop on our steps
Well it's to of course spend the reward to the Watchlist.
That's the last thing that needs to be updated we updated all the list here except for the reward.
So we're going to do that right now we take our rewards and we use your brain function to append the
last word that was just received perfect.
And just before we get out of the for loop we just need to do one last check to check that if it's done
then we want to stop the expiration.
And so we're simply going to add here a break.
Meaning that if it's done we stop the exploration and we directly move on to the next step which will
be the update of the shared model and now we are done with this for now that the agent has done its
exploration.
It will update the shared model and we will take care of that in the next tutorial and so then I.