Hello and welcome back to the course on artificial intelligence.
In today's tutorial we're going to talk about an add on that we're going to be implementing for our
8:3 algorithm.
It's called The Long short term memory or the LSD for short.
So let's have a look at what we have so far and then we'll discuss why we need the LSD and what what
else to him is so so far we've discussed through shall we.
And we've talked about all three letter A's in the 3C and of course we've seen that it's actually a
bit more complete or much more complex than what we have on this image we actually have three or more
multiple agents going through the environment and they're communicating between each other and so on.
But for simplicity's sake for today's story we're going to just illustrate everything with this one
agent at the end we have this report the critic part.
So basically once we have a state let's state to this image as it goes through convolutional there then
goes through a pulling Lego shuffling of flattening a layer.
And at this point we have values or numbers which are then propagated through the network and they go
into the hidden layers and then as the output we get the policy or the actor parts and they get the
value of the state or we get the critic part and what's we're going to do today is we're going to talk
about this hidden part in the hidden layers we can actually take it to the next level and we can add
a notification.
And we've already seen that multiple notifications exist for they shall move seen one of them we've
seen that in some cases you can have this main part of the network which is individual to every agent.
Or you can have this main part of the network which is shared and that's what we saw that in our previous
intuition trials we had a shared part of the network.
This network was shared between the agents and as Adlon will tell you more in the practical tutorials
that really helps with the challenge of breakup.
And there are lots lots of lots of other ways you can modify the algorithm.
Lots of other additions that can be implemented.
And one of them we're going to discuss because we're actually going to have in the practical side of
the trolls here before you hit and Lares which you can add is an elist Himmler a neural network Klare
which allows your valuation algorithm to have memory which allows the algorithm to remember what happened
before and we'll talk about LACMA just now in more detail.
But basically you can add an extra layer here which is elist him Lehre and enhance your algorithm with
some additional Nimer of another feature and which you will actually see in the practical Sorrells is
that we didn't even need any hidden layers after they all similar So you'll see that in Atlanta implementation.
He's got the flattening there right away.
After that he's got the stemware So basically this box represents the elist gambler.
And then after that right away you've got the output so you don't even need any any other hidden Lares
after that simply because that's how much power the LSM lair adds to the algorithm.
And again the algorithm or the architecture of your your own network it's a very visual thing it's a
personal preference.
It's a very creative thing so you might want to have to elist gamblers you might have well-learned analyst
Jim Lehrer.
That is several like five hidden layers after they list them.
That's totally up to you and for you to experiment and explore.
But this is what we came up with in the practical tutorials so you'll see that we have a flattening
flat layer and after that we've got an ls dam lair and then the output.
So now we've talked about the other similar so much what is this a handler.
Well the LACMA lair adds memory gives a feature like allows the neural network to have memory about
what happened in the previous iterations and it is often symbolized or shown with a symbol which looks
like this.
This is just getting started into it and I'm just putting here.
I know it looks very crooked but I'm putting it here so you can see when we further discuss this image.
You can see what's going on so this the output of this letter goes in to here and that is our.
So this is a whole layer going in here so it's a vector of values x is a vector avows goes into the
stem which we'll just discard the cell.
And then as an output you get another vector which is you know the concatenation of these store or some
Somehow it ties in our case as an output you get this and you get this.
So let's have a look at this in more detail.
So it's going to focus on this part.
In fact we get into as you probably noticed by the letters being on the side we're going to turn it
to the side.
So like that and the whole this whole like jumble around was just to reiterate the fact that even though
it looks like this what is actually happening is a layer of values a whole vector of values is going
in here.
Something is happening which will just cause just now and then a whole vector valises is going on.
So this is the layer it's not just one element of the this is the layer itself.
So let's go back again just to reiterate Lehre goes into this where something happens Lehre comes out.
So that's that's the LACMA is just on its side.
So it's just easier to just this way and that's a common representation.
Right now there we agree why this image was on its side and how are we going to proceed with this.
Let's start digging into this LACMA situation a bit more.
So what happens inside Ellis Jim Lehrer.
So this is what it looks like.
And of course this is looks very complex and we're definitely not going to go through all of this right
now simply because there's quite a lot to discuss.
The point was Operation Xolair wise operations and just there's just a lot going on or a lot of intricate
details which we're not going to go into because otherwise would blow out this of course and this is
not the purpose is not to talk about Else dams here which are going to utilize the LACMA.
And if you'd like to learn more about Ellis systems you can either go to or we're here.
Christopher Ola's blog.
He's got a good description of his stems or we also talk about LACMA in our deep learning age is of
course you can check it out.
We've got a whole section on recurrent neural networks and systems as well.
So basically this is the internal part of the system.
And what happens is like the leg goes in so we're going to talk about this on on an intuitive level
on a very basic level just just what was going to be enough for us to understand what happens or why
there's memory.
And so that you can also better understand what Atlanta is talking about when he's implementing this.
So Largo's in all of this is something something basically goes on here Larry goes up.
What do we need to actually see is that there are these out these parts there's actually additional
inputs into this Lehre.
So remember this usually you have like an input from a previous Lehre then this letter and then you
have an output if you think of that image we had previously the normal network which is not not on its
side which is like from left to right from top to bottom to top.
But unless you actually have more inputs.
So I know it's getting even more complex but these things at least we can understand them.
So this is your memory cell.
This is the key and this is what you'll hear Heidel Atlanta talk about.
So the memory cell is something that is saved in the elist him so these inputs and outputs are actually
here what you're looking at is the time axis.
So this is unraveled in time so in one specific iteration this happens but then this value is taken
from the possed and this values passed into these values these values are taken from the past and these
values are passed onto the future and how they pass wealth through the way the else teamwork so we're
followed worrying about too much what's going on here.
All we need to understand is that when the letter goes in and here we really have a value which came
from the past which is stored inside the LSD inside the long short term memory.
We've got this memory cell and whatever value was here before it is just it just stays here as you can
see it just goes through it flows through freely except for these pointwise operations where it can
be either closed off or something can be added to it.
But regardless of that it's just some value that flows through freely so it's basically it's passed
onto the next point in time next point.
So you could just think of it as a like some memory that like a flash drive or something like that that
this cell has and so just remembers the previous value that was in here and then it can use that to
do to either add to it or read from that valiance on.
And this value is the hidden state.
So hence the H and the hidden state is basically.
And now the value that comes from the past and then is used inside the system.
And as you can see at the end after all this happens what you get is you get a letter that comes out
and it is so that you get this value that comes out and it is the same value that's passed forward.
So basically the Ellis team remembers two things there's a constant value that is just like stays in
the list and that can be changed like this is a flash drive for like a constant value.
So the memory cell.
And so you can you can you have the luxury of storing something in that space and that memory and it
will be passed onto the future so whenever in the next iteration.
So like the algorithm was in an environment it saw something did something and so on.
And then in the LACMA you can store a certain value then it will remember this value even when it's
in the next state.
And also the other value they'll remember ill remember its previous output.
It automatically will remember its previous output so the output goes here and goes here.
So that's basically the very very very high level of what happens in an LSM.
Once again if you'd like more details as lots of resources where you can find and at this stage we just
don't need to go into that much detail on all of these things.
We just need to understand you know what a memory cell is what adherence to the memory cell is what
a head of state is and how like how the facilitates memory for him.
And the question is so now that we kind of have a general overview of all of this to reinforce or to
solidify this knowledge you're kind of like give a reason for this knowledge.
Let's ask the question why do we need memory.
Why do we need memory in our A-3 or other algorithms.
Well let's look at our example the challenge that we're taking on in this section.
So the challenge is breakout and what happens and break will and break out you've got this environment
these little blocks that you need to destroy with this little ball and you need to make sure that this
is your kind of like racket or platform that's moving around.
And it must wherever the ball is flying it must catch the ball and bounce off the platform and go back
and hit balls of the walls as well.
Go back a block and come back.
And so that's the essence of what you need to accomplish.
But now let's look at this ball like imagine you're a and and 83 C algorithm dure or an agent inside
one of those agents inside 08:30.
You see this picture what do you extract from here.
What would your action be here for you.
So you can see the balls flying right.
So well it is flying right.
So it's going somewhere and maybe it's flying towards the right you can.
Could you make this conclusion could you like anticipate that it's coming towards you.
You probably could and maybe you're in the right spot to catch the ball.
But what if the ball is actually not flying that way but is flying that what if it's flying that way.
The thing is you cannot tell from this one image which way it's flying because you don't know where
it was in the previous moment of time.
So if it was here then it's flying this way.
So if you had if you knew the previous moment time if you knew that it was here you'd be there now you
know here is as a human you just draw a line for these two and you will says going this way.
But if you knew it here you draw lines just as going this way.
Moreover look at this.
It could have actually been somewhere like over here.
Maybe it's going up maybe it's actually going that way so maybe it was here and I was going up with
so just from that one image is very hard is actually impossible.
It's just like geometrically impossible to tell which way the ball is flying.
And so that's why the LSD the memory actually really really helps our Mfat the memory it can still you
can still do a good job but could probably guess or you know find other ways of understanding where
to go.
But with the stem move just even that one memory so if we go back even with that one value that that
was kind of like the output of the previous value or maybe you know maybe the you can store it here
or based on this value or based on the information it gets from the previous point in time.
So let's say from what happened here.
So that's where your ball was before so you can pass on information about the environment from the previous
point in time through here.
Then now you have it now you know not only have your information from the image.
If we go back even further you'll remember that information from the image.
Well this is doomed but we're actually working to break out information.
The image came here here here turned into these flattened values.
And so that information for them.
Image coming into the whole system.
And now all of a sudden as as you remember coming from from not from somewhere but from the previous
point in time.
So that's where you of actually demonstrate come from top or from the ball from left right.
Actually it's just it's just stays in the elist him lair.
You have that information just through the architecture.
They'll say you have information about what happened previously.
And so we go back that information here helps you now make a decision on what to do.
Helps the algorithm make a decision.
And now all of a sudden it knows that.
OK.
So the ball is actually lying in either.
Let's say it's flying in this direction or in this direction so I'm in the right place I should stick
around here the ball is coming in my direction or if it or if it realizes the ball is lying there it
should start moving to the left because if it waits a bit longer it'll be too late.
And they will miss the ball.
So basically that's how the elist them where really helps in the algorithm and that's exactly what we
will see when you're doing the practical tutorials of Atlanta.
So there you go that's how these teams work.
And just one additional note as we mentioned at the start Ellis teams are not 100 percent necessary.
They're not a complete They're not completely attached they algorithm.
You might want to have them in a through C algorithm you might not want to have them depending on the
situation to bring in Arctic's you choose.
There are lots of additions and we've already discussed the addition or the modification where the neural
network is shared between actors are not shared between agents or not.
Now nonetheless elist Jim there's another one that you will see in the practical tutorials where we
add entropy which is calculated through a policy lösen Adlon will walk you through that.
So basically there's lots of different modifications that can happen in an A-380 algorithm.
Just remember that it depends on what you want to achieve and it's also something that would encourage
you to explore if you're going to be implementing lots of these and trying different algorithms.
We've already discussed a couple and maybe you can find some additional modifications that might be
of interest to you or maybe when you're watching these tutorials maybe buy them more modifications have
come out which are very interesting.
So definitely that's something that you could look into and that could further enhance your knowledge
of artificial intelligence and how to create these algorithms.
And on that note I hope you enjoyed this tutorial and I'll look for you next time.
Until then enjoy.
I.