Hello and welcome back to the course on artificial intelligence.
I hope you're excited about today's tutorial because we are taking our very first step into the world
the I.
And today we're talking about reinforcement learning.
It's a very important story because it will underpin everything else is going to be happen in this course.
So let's get started here.
We've got a little maze and this maze is our representation of an environment and that's what we're
going to be dealing with in this course.
We're going to be dealing with certain environments in which our artificial intelligence is going to
be performing it's going to be taking actions it's going to be looking to beat these in my going she'll
be looking to win in these environments.
And here we've got an agent.
The agent is our artificial intelligence.
That's the person or that's the mind that's going to be navigating these environments and learning from
the feedback that their minds are going to be giving it in order to perform certain actions.
And so the way it works is the agent perform certain actions in this environment.
And as a result the state in which it is in will change so it might be further or closer or more to
the left more to the right.
It might have sort of the other parameters that describe it state and those parameters.
So the state is going to change because of the action takes and it will also get rewards based on the
action.
So every time it takes an action the state will change and it'll get reward.
Now bear in mind sometimes it might happen that it won't change the state the action won't change a
stay or there won't be a reward for taking that action.
In that sense it was.
But nevertheless the agent's going to keep doing that was going to be taking actions cheating the state
getting rewards changing action taking actions changing the state and getting rewards.
And by doing that process it's going to be learning about what was going to be exploring the environment
understanding what actions lead to good rewards and favorable states and what actions the two rewards
an unfavorable state.
And this is a very simplistic representational very global problem.
So if you think about it environments actually don't have to be just mazes.
It's not just about getting out of a maze or finding a treasure in a maze.
An environment can be pretty much anything in life.
So imagine you waking up in the morning and cooking an omelette.
So in order to make that omelet you need to go through certain steps you need to get the salt get the
eggs get the frying pans which to fire on and so on and it does sound like a routine mundane thing.
But it's become routine because you've done it so many times.
But in reality it's an environment where you're performing certain actions you're taking that you putting
the fire on you putting a frying pan on the fire you're putting all the eggs into the frying pan and
you put some salt on the eggs and you're turning over and so on.
So as you can see they are CRN actions actions which are taking in certain states and those actions
lead to certain other states and sometimes reward.
So for instance when you put the fire on and you wait wait wait wait wait you take an action of wait
wait wait wait too long and then you put the eggs in into the frying pan.
The rewards are going to be very negative.
It's all going to burn.
On the other hand if you do all the all the correct actions in the correct time so it's also very important
to understand that actions should be taken at the correct points in time.
So for instance putting the salt in the frying pan before you put the eggs in might not be the best
idea.
You might want to take that action of putting the salt into the frying pan after the eggs are in there
so that in a different state.
So it's important to remember that.
And at the same time so if you take all the correct actions in the correct order in the correct states
your final reward could be that you get an omelet which you can eat.
And so that's a very basic activity in your life but if you think about it it is actually an environment
and you are the agent going through this environment and perform a task you don't really need to learn
anything because you already know it pretty well.
But at same time you could learn maybe you could learn how to make a better omelet or especially if
it's your first omelet that you're making you're probably going to screw it up.
But you will learn from that because you will understand what actions lead towards states and routes
and anything else in life.
For instance even trading on the stock market and you know buying and selling and getting certain feedback
from the market in the sense of return positive or negative returns.
That's also an environment that's you participating in that environment as an aged.
Driving a car is also an environment where you can turn the steering wheel you can accelerate you can
break and so on and you're getting feedback from the environment and you know one of those feedbacks
is the policeman giving you a speeding fine if you're going above the acceptable or allowed speed limit
on that highway.
And therefore from there you learn that that's not something that should be done because it leads to
a negative reward.
So rewards don't have to be just at the very end of the process.
They can be throughout the journey throughout the process.
So those are a couple of examples.
And in terms of a I the simplest way to think of reinforcement learning is like training a dog when
you train the dog you to give it certain commands and if it obeys those commands then you give it a
reach you give it like a biscuit or something if it doesn't Abeles Kamaz you tell it that it's a bad
dog or you just don't give it a treat.
And through that process it learns what certain commands or what it needs to do what action it needs
to take in certain states and the states are the commands that you're giving it.
And based on that it will get some certain rewards of course in the world of AI.
It's not that complex.
You don't have to give the treats.
You don't have to have like a bag of biscuits with you every time you just give it a plus one or a minus
one so it's a huge advantage that in the world of AI we've created these AIs ourselves.
So the rewards that we're giving them if you think wow this is really cool rewards are giving them they
don't actually exist they're just a plus or minus one or plus a one or a zero or something.
So it's all nonexistence all imaginary stuff.
But at the same time it leads to great results as we can create these amazing things these amazing artificial
intelligence as by this amazing artificial intelligence by just providing rewards we don't really exist.
Plus and minus one doesn't cost anything but same time release results.
So very similar to real world.
And you know for example Dokes But here the rewards are digital and just numbers.
And with that in mind we can talk about about robot dogs I love this example so this is just around
in pictures not necessarily that exact robot dog you know that is trained through reinforcement learning
some of the robot dogs especially the older ones you'd have an algorithm in there.
And this is actually a good example of the difference between preprogramed agents and reinforcement
learning agent so you could have a robot dog which is preprogrammed to how to walk it will say.
So in the in the algorithm behind the dog in the software will say OK so in order to walk you need to
move your left leg forward left front leg forward then your back right leg forward then your front right
leg forward then your back left leg forward and repeat that action and you know that's that's the definition
of walking is a function inside this dog.
And then it might have you know how to sit how to stand and things like that.
Whereas in a robot dog that is trained through reinforcement learning what happens is you don't preprogram
it.
This is the key concept to everything here that you don't have any algorithm inside that is hard coded
into the dog.
Instead you have what we'll be discussing in the future.
You have this reinforcement learning algorithm which is told that OK so the goal is from to get from
where you are now not knowing anything to that to the end of the room for example.
And here are the certain actions you can take.
You can move your right foot you can move your left foot you can move your right back foot you are left
back foot so here all the degrees of freedom you can do you can move it like this you can move like
that so like a list of actions you can take and your rewards are every time you take a step forward
you get a plus one every time you fall over.
You get a minus one and that's all there is to it.
And then they just leave the dog and let it figure it out on its own.
So the dog tries to stand up it falls then it realizes that OK I shouldn't do that action that led to
me falling because every time I fall I get a minus one which is not good for me then.
So does the other action that helped him stand up and then it figures are just experiments experiments
experiments tri's things randomly and then figures out that it can make a step forward by moving its
right front foot and he gets a plus one and realize oh I should do more of that.
OK cool so it now learns that it should do more of this and less of that.
And through this learning process it quickly very quickly understands how it can walk.
And those those dogs that figured out on their own can actually sometimes walk better than dogs that
are preprogramed because really preprogrammed things we look at the real life dogs and or you know we
use our own imagination how to do it whereas a reinforcement learning dog can optimize things on its
own.
And because in AI sometimes it can get even better results.
And that's how they can train these robot.
The same robot dogs to play soccer.
You can train a normal dog to play soccer because you know simply the whole approach is different.
And it's not something that you know probably a normal dog has been trained to do or has ever done in
its process of its evolution.
Whereas a reinforcement learning robot dogs can very easily understand how to play soccer as long as
you tell them what the rewards are what the goals are what the possible actions they can take.
So that is how reinforcement learning works.
In general there's a quick overview of reinforcement learning.
I hope that got you very excited about was going to come next because it's a completely different world
compared to preprogram solutions a hard program hardcoded solutions where you have the if else conditions.
This is very different.
And we're going to be talking more about that.
In the meantime we've got some additional reading for you so if you'd like to have some supporting materials
Here's a great article which you can look and look into.
It's called simple reinforcement learning with tensor flow.
It's got ten parts.
The link is here and you'll find the full clickable link on.
In the course of resources by Arthur Giuliani's 2016 article and you can follow along this course and
also get additional information from that article.
But bear in mind that that article is tends to flow where as in this course we are using pi torche so
different implementation but implantations but at the same time you might pick up a few things here
and there that might supplement your learning that we're going to be doing in this course.
So great articles follow you in if you're considering following it for sure.
Still just in case.
Check out that that first part and see if you like it see if you'd like to read it a bit more.
And then we've got specific to this tutorial a border enforcement learning there's a paper by Richard
Sutton which is called reinforcement learning.
One introduction is the 1998 papers are quite old but at the same time you can learn a bit about reinforcement
learning some of the examples like that omlet example and other examples of where reinforcement learning
can be applied and just a general overview of reinforcement learning.
If you are looking for some additional reading and on that note we're going to wrap up this tutorial.
Can't wait to see you next time.
And until then enjoy AI.