Hello and welcome back to the course on artificial intelligence.
Today we continue our journey into the world every three see and we're talking about the synchronous
side of aither C so there we have our abbreviation of synchronous advantage.
Active critic.
And today we're going to find out what a synchronous here stands for what it means.
And let's go back a step.
Let's look at what we started this whole course for enforcement learning what it's all about that the
Asian is in a certain state.
They observe the state.
They make certain decisions they take actions in that state and then the state is changed so they get
into a new state plus they get reward.
So the reward for taking that action or some sort of reward which could be a penalty as well and they
end up in a new state.
And based on that now they take another action again.
They get a reward and end up in a new state and they take another action and so on and so that is the
basis behind all of reinforcement learning.
And that's what we've been using in learning in deep learning and deep convolutional keep learning and
that has allowed our agents to beat gradually more complex and more complex environments.
But now we're going to introduce a even better concept and even for take this even further level.
What A-3 see introduces through this and synchronous element is instead of having one agent attack the
environment.
They have three agents or whatever number of agents or several agents attacking the same environment.
And the key here is that's why it's called the synchronises because they're initialized differently
so their star inputs are different.
So for instance as you'll see from practical sources you set a random seed and you set it differently
for each of the agents.
And that way because their starting points are different.
They're going to first go through environments in different ways and then they're going to explore in
different ways and then in the next iterations are also going to explore in different ways.
And so for instance we have three agents.
You're all of a sudden you're getting triple the amount of experience instead of just one age and going
through and exploring the environment and trying to understand how to operate it in that environment.
You now have three or however many of them going through that and getting this experience and so there
so that each one of them is learning for this bigger experience and apart from being just giving a broader
range of experience it also reduces the chances of one agent getting stuck in a local maximum.
So for instance if one agent finds a way to beat the environment which is not the most optimal because
if it deviates a left to the right from that solution that it found it always gets like gets more penalized
it might get stuck in a local maximum.
It might just keep doing that thinking that that's the optimal solution where where it's actually not.
Well the likelihood of several agents getting stuck in that same local maximum is decreases over decreases
with the number of agents so the probability of one agent getting stuck in a certain local maximum might
be high but Or might might be a certain value.
But the probability when you have three of them of all three of them getting stuck in that local maximum
is much lower.
And as long as they share experience between each other they can help each other out so if one of them
gets stuck for instance it's in a local maximum and just simply think that that's the best and that's
the best that's the best solution all the time and keeps doing that.
Well as long as it interacts with the other agents So let's say this guy gets stuck in a calm action
as long as interacts with other agents through the way we build our whole algorithm through cellular
and they will help him out.
They will give him knowledge that actually you know hey you should explore this or he will be likely
more likely to get out of that.
And also overall the environment will know that hey even though this a great maximum these other ages
have seen better options and we should keep exploring because it looks like there are better options.
So in a in a very short kind of rough intuitive understanding that's that those are some of the advantages
of having these are synchronous agents for so you have more experience to choose from and to learn from.
You could get to the solution faster and generally speaking if there's a lesser chance of getting stuck
in a CRN local maximum.
So let's see how this all plays out.
In this model that we've built so far so is remember this is what we've gotten so far through the actual
critic and this is like we're all teasing this is so far as you remember from the first to turtle we
did introduce this you know we had this already even in deep ocean.
Q learning's So we just named the X now but now we've introduced critic but so far doesn't really make
sense what's the point of having this critic and measuring the value of the state or predicting the
value of a stage using the same neural networks or the same approach.
But now is this is this is the partner's going to start making more sense.
What we're going to do is we're going to replicate this because now we have multiple agents So if multiple
agents this is this is what it looks like.
So the first way of imagining it is now we have these three days well remember what we said about them
sharing their experience between each other.
So this is actually like right now they're all independent.
You have one playing the game and other than playing the game another play in the game.
It's like it's like launching your agent on three different computers you put three different computers
next to each other and you launch them and you know that's great.
Like indeed you like you'll get you'll get more experience you'll get more variety especially if they're
initialized.
So we can assume from here that they're ill initial always initialized before even though we have the
same picture here.
Are we going to know that they're actually initialized differently so it's not going to be like identical
training identical learning from this game.
And so even if you like you put three computer side by side and you launch them yes you're going to
have more experience because you're going to have three agents playing and also you're going to have
a bigger variety of possible solutions.
So that's true.
But the problem is that they're not sharing our experience among each other or not learning from each
other.
So they don't have that synergy.
They don't have the advantage or the extra power that they would get if they were compering you know
like how if you have if you have a team of people they work better together than each one of them separately.
So like in a team here you got one plus one plus one.
It's three but in a team one plus one to spawn and not three is like the three because they leverage
each other's strengths and mitigate each other's weaknesses and same thing here.
So if you put these two computer side by side yes you'll you'll have more experience memory and possibly
someone will get a better solution.
Another one that's great but it'll be even better if they start sharing that experience.
And how do they do that.
Well it's through this Wii that we calculate it so this Wii value that's the output of our network is
actually like that.
So they have this same every so every time.
All these agents they're contributing to the same critic.
They don't have separate critics they have a common critic and that's the key of how the actor critic
ties in with their synchronous.
So there's one critic that's watching us as they get experience.
So how do we calculate the Wii.
We've got to get the Wii through.
As you remember we can get TV through the values that we get so the rewards that we get through the
environment.
And so as the agents explore their environment they are calculi they're predicting the Wii.
Plus they have the Wii that they can calculate.
This is this is all all ties back into what we've already discussed in the previous sections of the
scores.
So they already have a Wii that they that they can predict like expect through the rewards that they
know that exist in this maze and that they've already explored and as they explore them of course that
that value can change.
But also they have the Wii that this is the output of the neural network so as they're going through
this they're going to be adjusting their neural networks in order to better match that expected.
So basically this is shared the critic part is shared between the agents and that is how they share
the information between each other that's how they are able to kind of see what is going on in the environment
shared with each other and then use that as we'll see further in the next part in advantage.
So use that in order to optimize how they're behaving the environment.
And the other thing to note here is.
So this was a through C.
This is like the core of A-3 see up to here.
This is a type of version of 08:30 But there's an actually even better implementation of this.
A through C which you'll actually hear I'd love to talk about in the one of the first tutorials and
the practical side of things and what he'll be talking about is how the creator of Pi torche actually
made an adjustment to one of the codes that was shared and get hub where he took all of these as you
can see right now they have separate neural networks and they showed the Wii that adjustment that was
made was actually to take all of these neural networks and put them into one take them and put them
together.
So ultimately there's only one neural network here shared among the agents.
So before they had each one of them had one neural network which were shared for the actor and for the
critic one neural network Shelfer actual for the critic one neural network share for accuracy.
Now they all have one neural network which is shared for the actor or critic actual critic x or critic.
And then the critic is here in common.
So let's see let's.
Let's move these pictures to the left here so make some space.
And this is basically the architecture or the structure that we're going to be using in the practical
tutorials.
I know that like this may sound a bit overwhelming at this stage but we've got one more to talk about
which is advantage and there we'll see it better in action.
How does going so we'll talk about the intuition in action.
But generally speaking this is this is what it is.
This is there's one network which each of the agents use or they share.
Basically what that means is that they share the weights the weights of the network are shared between
ages and when they update it they update the whole network not just their own network.
And then they have outputs they have like these actions for each agent and then they have the critic
that is shared which is going to be monitored.
So I know all of this is kind of like like there's a lot of stuff right now but hopefully it's slowly
coming together at least.
The main takeaway from here is that the critic because it's shared.
That's how the agents are able to make sure that they are cooperating together in order to get the result
much faster.
And then in the next tutorial we'll see even further how all of this adds up.
All of this comes together.
And for now there's like I would like to recommend or we would like to recommend you a an additional
reading.
So this is a blog by Jaromir Jansch.
It's called Let's make an A3 see implantations is actually two parts implementation and theory.
There's the link and it's very similar to what Adlon will be implementing in the practical side of the
tutorial so it's not specifically for this tutorial just not just for Sutro but it's for this whole
section.
Encouragement there some additional information some additional insights there.
And so that's that's why we're bringing it up here.
But nevertheless in the next tutorial we're going to start pulling all of this together.
Everything we've discussed.
And I look forward to seeing you next time.
And until then enjoy I.