Hello and welcome back to the course on artificial intelligence.
Today we're talking about the living penalty.
All right so here we've got all Belman equation and as we've been going through this course we've been
slowly making more and more complex so so far we've already added these probabilities in here.
And also we've added the discounting factor.
Now we're going to look at in more detail at this side of the question where we have the reward now.
Remember previously when we talked about how reinforcement learning works we said we have an agent and
it performs actions in the environment and in an exchange or as a result of that it gets a new state
and which is now in and a reward for that action.
Well so far in our example we've only been getting rewards at the very end either if we get to the finish
line or if we for the agent ends up in the fire pit he gets a plus one or a minus one reward.
But that is a very simplistic approach to reinforcement learning and in more realistic scenarios you
will likely have rewards throughout the journey not just at the very end you might have rewards throughout
the journey.
For instance if it's an AI playing a game and if for example it's like shooting somebody in doom it
might get points for killing that enemy or it might be a different other game.
If it overtakes another car or something like that just because of the rules of the game not because
of its way of analyzing the game but actually the game is structured in a way that it's reinforcing
its giving points for doing certain actions even before the game is over.
So Sinatras like that are very common and not just in games and also in real life and that's why we're
going to introduce something similar into our example a simplified version of that but nevertheless
a reward that is continuously given to the agent throughout the game not just at the end and the way
we're going to do it is by looking at the other tiles.
So right now we only have a reward plus one at the final tile and reward minus 1 at the other final
tile the firepit.
But now we're going to add rewards in every single time.
We'll add a very small reward will be minus 0.04.
And as you can see it's negative so every time the agent moves he'll get a negative reward and that's
what's called a living penalty because no matter where he goes he will always get this negative reward
except for these final tiles because that's the end of the game.
And so you can see the reward even on this tile is madness or a puzzle.
But that doesn't mean that he starts with that reward.
He only gets this reward.
And this is important to remember he only gets his reward when he enters a tile so whenever he he promised
an action he goes here then he will get this reward minus 0.04 and then he comes back to this style
he'll get another mind and 0.04 word.
And so the longer he walks around the more he accumulates his negative reward and therefore is an incentive
for him to finish the game earlier so quickly as possible.
And so now let's have a look at how our policy or how the agents policy is going to change depending
on what value we set for this reward.
So here are four environments and in each one we're going to explore a different.
We're not going to do the calculations.
We're just going to project the results and you will see that intuitively they make total sense.
So here we've got a reward for any step offer any for getting into any state.
Is equal to zero.
Just as what we've seen before here the reward is going to be Mei's 0.0.
For what we just did just now you know the reward will be at minus 0.5 or the level of giving penalty
will be mine is open fire so much higher you can see them here more than 10 times greater.
And here are the living Penhall it will be minus two.
So even more than the rewards you get for jumping or even less than the reward that you are the agent
gets for ending up in the fire pit.
So let's have a look at how the actions or the optimal policy for passing this environment will change
depending on this reward.
So this is our original policy.
And as you can remember we had these two very interesting and even a little bit weird a decision by
the agent but which totally makes sense if he can live for as long as he likes.
If you can just travel around for as long as he wants without being penalized for staying alive very
long.
He why not why wouldn't he just go into the corner here into the wall and just keep doing that until
it happens.
It so happens that he goes this way and then he will walk around.
And same thing here it's much safer for him to jump into the wall hoping that one of these will come
up eventually and then he'll go to the finish line anyway because by choosing these two actions he doesn't
risk getting into the fire pit.
Now let's see what happens if we add a reward negative reward for just being a life for making a step.
Move here you can see that instantly these two changed.
Now the agent doesn't want to jump into the wall.
He is more likely to risk getting to the firepit having a 10 percent chance of jumping in here but he
will go forward because every time he comes to watch here if he was going to be doing it here as well
every time he jumps into well he performs an action he ends up into in this state with an 80 percent
chance.
And that means an 80 percent chance you'll get a minus 0.04 reward meaning that a lot of the time he's
going to be getting this accumulating this negative reward.
Same thing here if he jumps into the wall waiting for that moment when he will actually be randomly
moved to the right.
If he keeps doing that he will accumulate this negative reward and that the result of that if you perform
the calculations you'll see that the result of that the expected value of that approach jumping to the
wall is worse than taking the risk of going forward and actually ending up in in the firepit.
So he changes his decisions in these two blocks to instead move forward and here move to the left even
know there's a risk of the firepit fire simply because now the longer he's alive the longer he will
accumulate this living penalty in the next environment.
Now we're increasing the living Pouncey to even a greater number Meinzer point five and let's see what
changes here.
So now you can see that compared to this environment.
The only thing that changed here is that this arrow is pointing to the right.
And what that means is that now it's no longer a good option for the agent or actually also this arrows
pointing was pointing the left and nozzles nose pointing upwards.
So now it's no longer a good idea for the agent to go around from here or go around all the way because
if he goes wrong all the way yes he's safe or there's a lesser chance there's no chance of getting the
firepit.
But at the same time or there's less chance are going to happen.
But at the same time he will accumulate quite a substantial negative reward as he walks around.
So it's just it's the path is too long.
So that forces him whether he's here or here to take the shorter route to get here even though he has
a much higher risk of getting into the firepit because as soon as he ends up in the square there's a
10 percent chance of getting to the fire.
According to his calculations.
It's just the expected value of this approach is better than the expected value of going around simply
because we've increased this living penalty.
And finally we're getting to the example with the living penalty of minus two point zero.
So here I encourage you to post the video now that you've seen how the policy has changed as we increase
the loading punt penalty.
I encourage you to pause the video and think for yourself what will happen in this scenario.
What do you think the optimal policy will be given that the living penalty is so high so all this supposed
video if you'd like to.
And now I'm going to jump into showing you the solution so in this case if you increase the penalty
to minus 2.0 it's so high remember that the penalty here is only minus 1.0 it's so high that the agent
just wants to get out of the game in any way possible even if it's just by jumping into the fire pit.
He will do it.
He will be like every time I make a step every time I end up in a new in in your state or every time
I make an action I end up getting a minus two reward.
So what's the point of trying to get to the finish line if from here will take me two extra steps.
I'm just going to go here and then straight into the firepit because that way my reward is going to
be less than negative reward is going to be as bad as in the case of just making additional steps so
you can see that adding this living reward and depending on the value of the living reward that we're
adding the results are going to be different.
And the agent is going to select different policies and that's basically what's how the reward value
can be is incorporated by the Belmont equation even when it's not just at the finish line or at the
end of the game but even throughout the game.
And again once again doesn't have to be on every single in every single state depending on the environment
itself.
It might be given to the agent at certain specific states not at every state but in our simplistic example
we're just using rewards at every given state.
To illustrate this concept so I hope you enjoyed today's tutorial.
And as you can see we've already made our Belman equation quite sophisticated and now it can be applied
to many different scenarios and I can't wait to see in the next tutorial.
And until then enjoy a I.