Hello and welcome back to the course on artificial intelligence.
Previously we had quite a strenuous and long tutorial on Margrove decision processes and hopefully you
got along well with that.
And hopefully I could explain things in a approachable and engaging way.
And today we're going to talk about policies versus plans.
There will be a quick and fun tutorial because now we're entering into a new world we're entering into
a world of stochastic search nondeterministic search when you're just not getting through the maze but
also accounting for random factors that might just hit you in the head when you're going through this
maze and you need to be prepared for it.
That's the world.
Our agent is living in and it's more fun but it's also more dangerous it's more it's less predictable.
So how is our agent going to behave.
Let's have a look.
There's our mark of decision process framework which is once again our favor Belman equation.
However the the more advanced version of the Belman equation we are working with.
So from now on we're just going to call this the Beldon equation.
And here we've got our maximal and Crucell action so the value of a state any state as is the maximum
across all actions an agent could possibly perform in that state.
And the maxim was taken from the reward that the agent will get by perform action A instate as Plus
a discount factor multiplied by the expected value of the new state it will be in.
And I'd expect those taken here because they are doesn't know exactly what sadle end up in.
They are some random effects that are present in the environment that might alter the state and not
might not end up in the desired state.
It might end up in a different state.
That's why we're taking the expected value over here somewhere here.
So let's have a look at this as our example our or in our example of the maze.
So this is what we had previously so previously we're dealing with live deterministic search.
So we knew that.
All right so if I'm here I definitely need to go here if I'm here.
I definitely need to go here if I'm here I definitely need to go here if I'm here I'm here.
So it was all pretty straightforward.
Once you have this map and remember called it we called it a plan.
Once you have the plan it's pretty straightforward to do.
There are.
So that's the plan with arrows.
And from here it was very straightforward we're this is these are the routes that they will take whenever
you start on this blue line.
That's that's exactly the way you'd go.
However now we don't have a plan anymore.
We can't have a plan because you know whatever we plan might not happen it's not under control or plan
is when you know exactly what you need to do next.
You know the steps.
So you have a you have a starting point you have a goal and you know every single step so you can plan
them out you like I'll do this one I'll do this one I'll do this like in life like a plan.
But at the same time there's so much now randomness going on.
You can have a plan because what if you get here and then you click to the right and actually takes
you down.
So that's not part of your plan.
So that's why it's called the planning more.
And here we're going to calculate the values are actually going to just look at the calculated values
for this same problem.
But based on that given that we have this randomness inside.
So these are the new values.
And so why are these values different so let's just compare to what we had previously.
This is what we had previously.
These are then you.
So once again we had previously because he won 3.9 percent.
He was really 366.
And this is what we have now a a a less than once in force and 1 6 3.
And by the way these are not exactly the current rallies off the top of my head but if we were to run
an agent some values would be something similar to this and the values could change because depending
on the gamble that it would choose 3.9 or other value but nevertheless for the for argument's sake these
are the values that we're dealing with now and they're approximate they convey the whole notion in the
correct way so let's have a look at them.
Why have they changed.
Well why is here with this one here the value was one.
Why is it all of a sudden 0.26 Why is it less than one.
Just go from here here.
Well we actually called because from here if we go right which is our intention if we go right we could
I would actually we have a 10 percent chance we'd end up here.
So we'd hit the wall and would be back in this state.
And remember we have a Gamla So the value would be discounted and or we're off or off at 10 and chance
would end up here in this state.
So it's not 100 percent probability I would get here so therefore disvalue can no longer be a one it's
something less and it's 0.26.
So that's an example of why it's like this.
And you could get the exact value if you calculated the Belman equation the full but my question that
we have now.
The only problem is that there's going to be some recursion because you would need to know the value
for this and then you need to know the value for this that is quite complex and that's why we're not
doing the calculations manually here.
That's why the I can do them as it's going through all this it's it's like it's like nothing too complex
for a.
You can't play these things.
So that's our value here.
But of this is a different one.
So here it just to be 0.9 just because of discounting factor remember from here to here again now from
here we colleges jump from here to here simply because even if we jump if we go like this we might end
up back here back here.
Right this 20 percent chance that will still stay in the square because we'll hit a wall.
And again and so on.
So the value of being here is zero point seventy one.
Again this and the discounting factor.
You know this might look odd to you that this is even with the discount in factor this is too high.
Maybe the discounting factor in this example is not 0.9 maybe it's seven point ninety nine or something
that so don't worry about that.
Just kind of like focus on that.
The values have indeed changed that the values are now less.
Mostly because it's not it's a hundred percent probability to get to the state that you want to get
and what you will find.
An interesting one is here that here just to be 0.9 actually has dropped very much has dropped substantially.
Why is that.
Well because if you go from here up which is our intention there's a 10 percent chance of hitting a
wall but there's a 10 percent chance of actually ending up in the firepit and losing minus one to reward
and basically that means for the agent that's that's end of the game.
And so this is a very bad state to be in.
So all of a sudden remember we had zero point nine years apart and so they were equivalent.
Doesn't matter you hear here they're pretty much equal in terms of value of being in each of these states.
But now all of a sudden bam this date is like nearly twice as good as this one simply just because here
if you go straight to it you go right where you want to go.
The you know the consequences of the randomness occurring is you just stay here.
Here one of the consequences a 10 percent chance is you end up in the pit.
So as you can see this is no longer such a good state anymore simply because of something that fluctuation
that could happen.
As you can see this one is also very bad because it's as bad as this one in terms of you know it's only
10 percent chance of ending up in the pit and 10 percent chance of ending up in the wall.
But at the same time there's a discounting factor So first of all the discounting factor and also after
this one you'd have to go here.
And even if you hypothetically went here you could end up in the pit again.
So that chance would also be taken into account because remember this values derive from this value
and this value is derived from this value.
Right.
And therefore it's small but in reality actually what I said there was wrong.
This value is not derived from the Fed.
So if you just have a look now you will notice that this value over here is actually greater than this
one.
You will notice that for the agent it's better to go all this way than this way and it makes sense right.
Because this way it doesn't lose it there's no chance of getting in the pit.
Yes is a bit longer and therefore the discounting factor has a bigger effect.
But at the same time simply because there's a chance of getting in the pit here if it goes straight
it will there's a chance of jumping.
So it will take a draw to take its time and just go around because that way there's a much lesser chance
of it getting But there still is.
So from here goes there it from here goes there.
It could potentially get into the pit because it could end up there and that could end up in the bill.
But nevertheless it's a lesser chance so it will just go on like that.
So very interesting to see how they're all change remember previous you from here you'd go like that.
From here you'd go like that and from here we go like that.
And now all of a sudden you can see his change.
Let's roll the arrows and see what it looks like now and voila.
You see even a more random thing right.
So yes this is true.
But look at what happened here.
Look at this one.
Were you expecting that.
That's something I definitely like when I saw this one first time I was was very impressed.
I was not super I was not surprised and I was not expecting this at all.
And this is an example of you know when I can outsmart a human.
It sounds like something you caught even you could predict but the I through enforcement learning remember
that example of dogs can actually sometimes work better than normal real life.
Dogs are preprogrammed robot dogs can play soccer simply because they come up with these ideas that
even we can't see.
And as a great example you probably weren't expecting that as well that the Asians instead of going
up is like why would I.
Like if I go up then there's a 10 percent chance I'll jump into the pit.
But what does it achieve by going into the war.
Well 80 percent of the time will bump back and stay in the state.
But 10 percent of the time will go here and 10 percent of the time I'll go here.
So all of a sudden you can see that now it's actually in this new approach of jumping into the wall.
There is a zero percent chance it will go into the fire but from this spot so.
And it's like it really doesn't want to go into the fire pit so drugged bon bons into the wall a couple
of times and then it will go the right or left at some point because that randomness is going to happen.
And so it learned that through experimentation it learned that OK when I go forward the results are
not as good as when I go to the wall.
And if you think about it it's like this.
This robot if you think about it this is a firepit is a very this is the this is like a square is like
a very tiny ledge and then this is like a mountain like a cliff.
And this robot is just hugging the cliff and just like trying to waiting until it like pushes right
or left because well as a human you probably do the same you wouldn't be standing facing that way or
you'd be hugging the cliff right.
Or something like that.
And hopefully you know we need to end up never end up in situations like that.
But like visually just visually if you think about something here.
And so that's pretty intense right.
So that the AI came up with this idea and same here that is sort of going left and Riskin get in a fight
but I'm just going to try balls off the wall like you know hug a wall try to jump into the wall and
at some point I know that you know just there is a probability is a 10 percent chance every time I do
that I'll go here and something will happen and I'll end up here and I'll be safe and then I'll just
keep going like that.
So very very interesting approach that they took here and you can see the the routes are like this so
from here it might go right and then it'll go right to the exit or here or go left like that.
And here we will at some point you will go left and go like that again.
This is important.
I'm not a policy so even when it jumps from here it will go here.
Maybe And then from here it might actually rain straight.
It might actually go back to the right and then from here and I going to let me get that right.
So there's lots of different options for it guys who might not follow exactly this ironmonger go the
other way.
This is just the desired routes that it's designed for itself.
But the way it'll work out is actually could be different.
It depends on the real world.
So there we go.
That's the world of artificial intelligence.
That's what a policy versus a plan is and hopefully you're getting slowly getting excited by what the
AI can do especially given what we saw over here.
These are some very virtuoso type of decisions that the AIs coming up with.
And as you can see when you're playing AI even from this small example you can see that even when you
play in a real world maybe you'll come up with ideas and decisions that even sometimes humans can come
up with.
And that's exactly kind of like what happened in those games where the Google Alpha goal was playing
versus Lisa idole champion of goal in Korea back in the world champion of go.
And they were playing in Korea back bakla in 2016 I think is March 2016.
It came up with some moves that humans had never played in 3000 years or humans were not used to playing.
And this is this is exactly an example of that.
So once again hope you're getting excited and pumped about discourse and about what we can integrate.
And I look for it.
See you next time.
Until then enjoy.
I.