Hello and welcome back to the course on artificial intelligence.
In today's tutorial we're going to have some fun.
We're going to have a look and artificial intelligence actually going through that maze that we've been
talking about so long and is going to use kill learning to navigate its way and find the way out and
we'll see what happens to the q values was going to happen to the policy and so on.
So let's have a look.
We're going to be using some materials kindly provided by the Berkeley University.
So if you go to a I don't Birk only the E R K E L E Why don't you just go to that link again.
You'll see this Web site and hear what we're going to be looking at is the need to go to we're going
to go to PacMan projects.
I think Pacman projects and here if you scroll down and you look at them in first learning this is what
we're working with.
So here you can download the zip archive.
So that's if you want to.
So don't you don't have to this is we're not going to go through a solution together in this trial just
letting you know where this is all from because we're very like.
We really appreciate that.
UC Berkeley has made these materials available.
But if you do wish to experiment with this on your own.
Just bear in mind this is not part.
Is not going to be part of our courses as part of the Berkeley course.
I'm not sure how it works for illustration purposes but if you do want to experiment with this you can
find it here the zip archive and all old instructions as well.
And we're just going to go into Python right away and first thing I wanted to show you is that here
we've got the licensing information so this is what I mean.
We're very lucky that they said we are free to use or extend these projects for educational purposes
provided you know distributing publish solutions which we're not going to do.
You retain this notice which we have and you provide a clear archbishop to UC Berkeley including a link
to which we also have.
So once again if you'd like to learn more there the link.
You can have a look.
And thank you very much all of these people who have worked on this project so here's the grid world.
We're going to be working if there is a solution there.
You would have to in order to make it work you'd have to solve it yourself or possibly find a solution.
Maybe some of your some people somebody you know might help you out with that.
If again what you want to you don't have to because we are just going to look at it on this screen right
now.
So after we've created all those files we could just launch it over here.
So there are some parameters that are involved in this whole world and we're not going to just show
you what it looks like if we launch it.
So let's try to launch it in manual mode.
So if I go minus one of these panoramas are manual so I can command your control agent.
So here you can see all grids so I can go up up so you can see that it's taking action starting and
started in states where I was.
And then you saw that I pressed up took action Norf and first time I ended up in zero once I did go
up.
But second time I took action Norf and I ended in the same sad didn't move.
So something happened you know the randomness happened I either went left or right.
And by default the parameters are set.
You can see here by default they're set to exactly what we discussed that how often actually results
in unintended direction.
20 percent of the time to 10 percent to the left some to the right.
So if I go up and say I went up I go right.
I went right right now didn't happen.
Right again and right and I'm finished.
But in this implementation you have to click again to get out of this final output so out of there just
click again and you're finished.
That's a terminal state so we can run our manual.
You can see that if I go right right right left up.
So here what we saw previously that the agent wouldn't go straight up right.
What's the point of going up if there's a chance of going into the pit.
So let's see what the agent would do.
It would go left and go west here would go West.
And you see I clicked left but it went up and here I would click right.
And I end up in the final exit stage and you see God's reward equal to one.
So that's what it looks like manually.
Now let's actually hook up an AI to this and let it go through.
So let's do an H here and let's add some Brandner.
So let me just see what I typed here so hopefully you can see by grid world why then here minus our
means.
That's the reward for living.
So I've got two of them so I probably should remove this one.
So minus k is how many iterations.
That's way too many iterations.
Let's do less Let's do like 10 iterations should be enough.
Minus a is Agent.
What type of agent don't want to do honor and image and some value or a Q.
Q So I want a Q.
Q learning agent doing this minus s is what is s speed so that's way too large a force that just use
the full speed for now minus R is a living penalty's so by default is zero.
So remember at the very start restart 0 living penances so let's call it also 0 0 and can just remove
this parameter and D is what is d discount.
So I just kind of factor so let's keep it at zero point and so very similar to what we're starting off
in this section on the course so let's run that OK way too fast again all actually so pretty OK so you
can see how he's exploring.
And so so far he's hit negative three times and you can see how the q values are being updated in these
squares.
So these are key values.
They are sort of zero.
You can see now the Q value.
So he's learned that this one is a bit different implement because once you get to the final stage you
have to get out of it.
You have to just click one more button to exit.
And so it's very close to one but not exactly one.
But at the same time you can see that here you know the value slowly kind of crystallizing hands are
a point a ex-colleague getting somewhere but they're just so far they're kind of zeroes because he doesn't
have enough information to understand what's going on.
OK so let's see let's see what happens here.
Exploring exploring exploring what's going to happen.
Well was being a while.
And we get this some randomness involved here.
So there is that good one a few times.
Now he only gets 10 iterations.
So he's got to learn fast Ok I need you there.
Let's see what's going on.
Come on.
Get out of that maze already.
And yes 10 episodes so average it turns out that.
That's not really interested in that.
So here let's see.
I've never seen enough of a click.
Right.
There we go.
So you can see this is the policy that he came up with.
Even through just 10 episodes he's already got a pulse.
I'm going to go up a bomb and here I'm going to go down here I'm going to go down here I'm going to
go into the wall and then I'm going to bounce we're here.
That's pretty cool.
OK.
So now let's increase the speed.
What was the parameter s there.
And that's like double lawlessness.
That's quadruple the speed and let's increase the number of iterations so let's say 20 to ration this
time and let's see if he can get through a bit more now.
So you can see he's going a bit faster.
And he's learning he's learning that it's not really you know out of this state there's not many good
actions Orio these actions that the right and straight are not that good.
Definitely this was definitely not good.
He still needs to learn that so from here is also good.
You can see that this action is pretty good.
All right.
What did he get.
OK.
So interesting policy here you we decide to go up.
Just not enough information.
So let's let's really do that.
And let's increase the speed to like 100.
Super fast and the number of iterations will give him 100 iterations this time it's run that scene like
crazy fast and you can see that because there's so many more iterations He's got more information more
opportunity to experiment and to actually build out this this matrix or matrix these values for every
single state.
He now knows you can see that zero point eighty nine.
What did we say in our zero point 86.
Another thing to remember is that the value of any given state.
Remember that formula we had is the maximum of the cube values.
Remember that thing that we came up with shortcut formula.
So what is it what with the value in this state be the V of this.
It would be 0.18.
Because that's the highest out of the four here the value of this state 0.7 you want the value of this
day.
Is there is point sixty one and so on.
So that's something to remember.
I remember when I was up I think we had like zero point 86 or something so praecox.
And so if we go next year I'll just disappear or disappear again and this can make it come back.
OK.
Slowly slowly slowly filling up some spaces.
I see.
And it's also pretty random because not only the environment has randomness but also the way he explores
that the star really doesn't know the policy is he's exploring at random.
Just keeps disappearing.
I don't understand why.
Anyways let's see what happens if you increase the number here and here should pretty much take the
same amount of time if the speed doesn't have a cap on it.
OK so he's like he has more opportunity to explore things.
OK let's see how it all goes.
And you can see the values are converging they go up and down depending you know because there is some
randomness and he might end up like in the pit even though he goes like this way.
But at the same time they're slowly starting to converge to some sort of values and cue values.
OK probably a thousand is a bit too much in terms of time.
It doesn't look like the speed is proportionally increasing as well.
So it might cut that part.
I mean like reduce the speed.
You know while this is very low you don't have to watch to the end of this tutorial I just want to experiment
with quite a bit so to give you some like examples of what we've been working through but you get the
point that it goes through all of this.
It has some randomness like Rambler's built into his behavior.
So even when it has like a policy it will still continue exploring so it won't just like once it has
a basic policy it won't just continue following its policy.
It will still experiment with other variations once in a while in order to enhance its policy maybe
it hasn't found the best policy already right away.
Maybe it can improve the policy.
And that's why even after so many iterations you can still see some random effects it is sometimes jumps
in to random states not just because of the randomness in the environment but also because there is
some level like a parameter which you could control which you could set up for your agent saying that's
you know most of the time 80 percent of the time do whatever your policy tells you to do but 20 percent
of the time you just have some fun experiment and see what happens and use that information that you
gather to update your policy.
OK this is taking way too long.
Let's try that again.
Yeah so that's how the agent learns in different states.
Maybe let's just run one more just out of curiosity.
So is there anything else we can change about it.
Iterations.
OK.
OK let's have a look.
Yeah well we could change the discussion for example.
So in this case we could say K minus a hundred minus a Q minus two and minus are OK thousand.
So reward.
We want to keep it maybe let's keep it at 0.04 But let's say set against this keep the reward at my
desert point zero for every time.
And then here we're going to say that the discount is not zero point nine but it's like zero point point
five.
So it gets discounted quite a lot as you go through the game.
So it actually now will be incentivized to be closer to the finish rather than further route the states
close to finish will get a high value so you can see that the values really drops off it's not as as
green as it was before.
So here you can see that this is the policy now.
So it goes like that like that like that very similar to what we saw before just probably only differences
from here jumping straight into here.
So that's one.
And OK let's just run one more.
This is so much fun.
Let's just run one more so k minus k 100 a q discard.
Keep it as it was original So let's just run this basic vanilla set up ok ok ok.
It's going to see if it will show us the policy.
And yes we got the policy.
Yes.
Good finish.
So here we've got the policy.
You know this is familiar.
Remember that time when we saw that the AI outsmarted the human bomb into the wall to go there and boom
into the wall to go like that to increase the problem.
So there we go.
That is an example of artificial intelligence inaction very very basic simple kill earnings so no deep
learning at this stage.
But at the same time it's already pretty smart and I hope you enjoyed today's tutorial.
And once again thank you to UC Berkeley and I hope you enjoyed today's tutorial and I look forward scenics
them.
Until then enjoy AI.