Hello and welcome back to the course on artificial intelligence.
Today we are finally talking about Kule learning.
All right so we've already got this equation the bellmen equation which we've added lots of components
to.
We've got the reward here which can be not just at the very end but it can be at any given step.
We've got the discount factor.
We've got the probability because now we're looking at mark of a decision processes.
And here we've got the possibility of ending up in a different states regardless of what action we take
or actually given the action we take.
There can be multiple states that we can end up in and then we got the value of the next states because
he kind of like a recursive function and so on but you probably still have one question.
The question is where in all of this is no letter Q Why is it all called q.
Learning.
So where's the cue.
And that's the question that we're going to be answering today.
So far we've been dealing with values the value of being in a certain state and now we're going to look
at how Q fits into all of that as well.
So here we've got two examples on the left is what we would be do so far our agent has been analyzing.
Ok I'm over here.
This is a mark of decision process so doesn't matter how I got here.
The rest of the environment doesn't care of the steps that it took me to get here from now on.
I have to make the optimal decision where to go here here or here.
Based on the current state and all the future states that come from here but not from the past.
And so he can see that there's three options there's state one state to state three.
And based on his experience he has calculated the values in these states and now he's going to using
the bellmen equation.
So even though this is a classic Proceso he knows that he'll go here but there's a chance that he will
go left right and so on.
So based on these values going to make a decision that's what we do so far and that is totally legitimate
approach here.
But now we're get modified a little bit.
We're going to take the same exact concept same exact problem but here instead of looking at the values
of each state that he can end up in we're going to look at the values or the value of each action.
So we're we're not going to use the letter V anymore because for the value of the state we're going
to use a Q and the you might have a question why the letter Q Well.
Q Some people speculate that.
Q Will I read this.
I think on Quora.
Somebody mentioned that Q is because of quality but at the same time I couldn't find any other references
to that so it might not be because that might just because that's the letter that was used at the time
and now it became super popular because it's all called key learning because of that.
So no exact reason was hold.
Q But nevertheless at least it helps us distinguish between V and Q So Q here.
There were presents rather than the value of the state it represents lets go of quality.
It represents the quality of the action that represents.
OK so I've got four actions.
What are the different qualities of these action.
What is the value of the action or the quality of the action which action is more lucrative so I need
a metric telling me all right how do I quantify this action and then I can compare them and that is
exactly what Q is.
And so he's got four possible actions.
As always go up right left or down.
And based on the action theres going to be a formula which tells us the quantifiable value of that action
which we're calling the Q q value of that action.
So let's have a look at how we going to derive this formula.
Q What how does it actually relate to these.
Because as you can imagine because actions lead to states there has to be some sort of link between
the two.
Right we've got we've already determined how to calculate this and we're pretty good at it.
We know how to use the Belman equation in very different environments with lots of different complications.
Well let's leverage that knowledge to understand how we can now calculate.
Q In order to make the same predictions because as you can imagine the environment doesn't change depending
depending on what approach we use the environment is going to be the same regardless.
So therefore this approach and this approach should always give the same result and therefore that's
another reason why these two should be linked.
So let's have a look.
So here is our view approach where we just get to look at the value of any given state this state or
any other state.
And here we go into we're just using the lead here because that's the current state.
And so therefore the terminology will be the same in both equations and here we using q as a Q Is the
of the state s and the action.
A because action is up but in which state we perform that action we perform that action in the State.
OK so now we're going to ride out the Belman equation for the first approach as you can see here we've
got the of s or the value of any given state s is the maximum of the reward that you get a maximum bet
based on the actions you have three.
In this case you actually have four actions so maxim out of all the possible actions of this part which
we've heard discussed many times so this is our reward that we get from performing that action in that
state plaza discount in fact multiplied by the expected value of the new state that we're going to be
in an expected value because it is a stochastic process.
We don't know exactly for sure that we're going to end up over here we might end up on the left or the
right sort of probability.
That's why these probabilities are in you.
All right so that's our value.
And now let's look at.
Q So Q is going to be defined.
We're going to use this to define Q So let's say the agent from this location from this state perform
the action up.
What is the q value going to be called to.
Well first of all let's see what he will get in return for performing this action up.
First thing that you'll get is a reward right.
Knows no doubt about it.
There's going to be some sort of rule or might be zero but we know that the whole is the way this reinforcement
learning process works is that some towns are performing certain actions from a given state or two.
So I'm going to add that in here.
And then we're going to add what are we going to add.
Well let's think about it.
What is the next thing that happens after he's going there.
Well next thing that happens is that now the agent is in a certain state he could end up here with a
80 percent probability or some probability.
But actually up here right here.
But wherever he ends up now there's we already have a quantified metric for that state he's in.
And that is actually the value of that state.
But because he came up in many different states and three of the possible different states we have to
look at the expected value of the state that he'll be in.
And so we're going to add that in we're going to add.
Of course the discounted factor as we previously had because that is somewhere in the future.
And then we're going to add the some of across all possible states across all possible states that he
could end up by taking this action.
Terms of probability.
So what we're saying here is that OK so by performing an action you're going to get a reward Plus which
is a quantified metric Plus you're going to get.
You end up in a state we don't know which one it could be here.
Could be here it could be here.
But here is the expected value of the state that you're going to end up in.
And now we're going to multiply by discounting factor because that is one move away.
So that is our Q value for this for performance section and what you will notice here right away is
that.
Q The Q value is actually exactly identical to what's inside these brackets over here.
And why is that.
Well if you think about it here we're taking the maximum of the results will get the maximum across
all possible actions so we got for action taking the maximum across all possible actions of the result
that we'll get by taking each of those actions and enqueue we're defining.
Interesting.
What will we get by taking a certain action.
So if you think about it it makes sense that the value of a state.
So for instance this state is the maximum of all of the possible Q values.
Right so here in the States by being in the state the agent has one key value to keep the 3Q value for
q value.
So yes positive for possible Q values while the value of the stay it makes sense that the value of the
state is the maximum of all of those four key values.
That is exactly what we can see here.
That's a good confirmation of this new formula that we derive.
If that wasn't the case if that if that didn't match up then we would have questions would be like.
So why why doesn't it match.
Why doesn't it match up if.
Q value is a quantified metric of performing an action and V depends on the floor.
Is like is the maximum of the possible results of the four actions that he can perform over that makes
sense.
And that confirms the formula that we've just derived and now we're going to make it even more interesting.
We're going to get rid of the Wii entirely because you can see here you've got Wii is a recursive function.
So and then you've got me and then B and then B and then B and so on.
So you can express this view through all of the following Vee's the most optimal these will come up
here.
We're expecting Q As a funk a recursive function of the OR as a function of the next V and then you'd
have to plug in this V and then we get back to the B.
So what are we going to do is we're actually going to take this V and we're going to we're going to
replace it with Q Right so let's have a look at that.
We're going to take this V of the next state and we're going to plug this into that formula over here.
And as you can see now so this part doesn't change this probability doesn't change.
But as we just discussed the of s is the maximum by all actions of q of S and a right over here.
So that's what we're going to replace in here.
So we're going to say a maximum of of course is the new action the action that we're going to take because
here we've got the Wii of as prime.
So here now we've got the maximal console at a prime.
So the actions that we're going to take from this state are from wherever whichever other state we end
up in but the action we're going to take from there and Maxima across all those and the maximum is of
all the cube values that will that are available to us in that new state as prime comma a prime.
And that's action.
So that's the.
So there's going to be another four.
Q values there.
So now as you can see let's go through again.
So as from what we derive this word would be just cause just through logic and intuition so that we
can see that VNS are actually view of AS and of and a are linked.
The of S is the maximum across all actions of Cuba S and you can see right here so this this part is
identical to this part.
And then we're going to leverage that and we're going to replace this bit with VNS from here but not
this exact funnel we're going to take this internal part and replace it with kill the innocent.
So we're going to plug that in here and this part is going to be q of s prime a prime maximum of cube
by Crucell a Priam's of Q As Prime a prime.
And now we have our formula.
So now we have a recursive formula for the q value so now the agent can think what's the value of the
section what's the quality of this section was the new value of this action.
Well it depends on the reward I get in the immediate step after that plus it depends on the discounted
factor times the maximum of all the possible Q actions in that state.
But I don't know if I'm going to get their side need to also look at that state in that state and that's
why we have this expected value over here so we have some probability times the maximum that's expected
value.
So a very similar formula as you can see but this time we're expressing things through the q values
and that's why this whole algorithm is called Kill learning because this is what is looked at this is
what the agents actually use they don't look at the states look at their possible actions and then based
on the actions on the q value of the actions they will decide which action to take.
So they'll just look at the maximum Q value in this given state it has four actions.
What is the best action to take so it can compare sort of comparing the different states that can end
up end up in is going to compare the possible actions that it currently has then by finding the optimal
one is going to take that action and then engage is going to repeat that process repeat that process
and so on.
So now you can see how all this comes together how the reward the discounting facts or the stochastic
market decision processes and the values and the q values all come together in order to cueist this
one super powerful Belman equation for q values which we can now apply and let our agents learn how
to beat the environment.
And so that is a intuitive explanation of what's going on.
I know we went through the formulas but it is necessary because this is like our formula that's we've
been going through this whole chapter and I think it's a good transition from the To.
Q And it illustrates how there are links between Yishun And if you'd like to get a bit more of a rigorous
approach mathematical approach and like you see the mathematics behind it and learn a bit more about
q values and how they work.
Then we've got some additional reading for you.
This paper is called Markov decision processes concepts and algorithms by martín von Autor low 2009.
So you cut the link here as always and here you can read in a bit more detail to understand all the
nitty gritty behind Hugh values and so on and now that we've discussed all of these things relating
to the Belman equation now we are ready to look at something more complex such as this paper in order
if if we want to get some additional information on this in order to kind of get a deeper understanding.
But even if you don't read the newspaper or radio you should have a good working knowledge of what learning
is all about and how agents come up with the actions that they need to take in a certain environment.
So I hope you enjoy today Statoil and I look forward to your next them.
Until then enjoy.
I.