Hello and welcome back to the course on deep learning this is an additional tutorial to talk about the
soft and cross entropy functions.
It is not 100 percent necessary in order for you to go through all of the parts that we've been through
in the main part of this section where we're talking about the convolutional neural networks but at
the same time I thought it would be a good addition to your bag of knowledge and skill set.
So let's go ahead and dig into these functions.
So to start off with what we have here is the conclusion of a neural network that we built in the main
part of the section and then at the end it pops out some probabilities for zero point ninety five for
a dog 0.05 five or 5 percent for a cat.
Given that photo on the left as an input This is after the train has been conducted this is actually
it's running and it's classifying a certain image.
And so the question here is how come these two values add up to one.
Because as far as we know from everything I learned about artificial neural networks there is nothing
to say that these two final neurons are connected between each other.
So how would they know what the value of the hold each one of them know what the value of the other
one is.
And how would they know to add their values up to one.
Well the answer is they wouldn't.
In the classic version of our artificial neural network and the only way that they do is because we
introduce a special function called the soft max function in order to help us out of the situation.
So normally what would happen is the dog and the cat neurons would have any kind of real values that
they don't have to be they don't have to add up to one.
But then we would apply the soft max function which is written up over there at the top and that would
bring these values to be between 0 and 1 and it would make them add up to 1 and 3 PPTA.
The soft max function or the normalized exponential function is a generalization of the logistic function
that quote unquote squash has a k dimensional vector of arbitrary real values to a k dimensional vector
of real values in the range of zero to one that add up to 1.
So basically it does exactly what we want.
It brings these values to be between 0 and 1 and make sure that they add up to 1.
And the way it works is that the way that this is possible is that because at the bottom we're here
you can see that there is a summation.
So it takes the exponent and puts it in the power of Zed and adds it up so that one's a two across all
of your classes.
All of these values.
And so there that's your normalization happening right there.
So that's how the Saucebox function works and it makes sense to introduce the soft next function into
convolutional neural networks because how strange would it be if you had a possible classes of a dog
and a cat and for the dog class you had possibility of 80 percent.
And for the cat claws you had a good 45 percent right.
It just doesn't make sense like that and therefore it's much better when you introduce the soft next
function and that's what you will find happening most of the time in convolutional and neural networks.
Now the other thing is that the soft max function comes hand-in-hand with something called the Cross
entropy function and it's a very handy thing for us.
So let's first look at the formula.
This is what the cross entry function looks like.
We're actually going to be using a different calculation going to be using this representation of the
century but the results are basically the same.
This is just easier to calculate.
And what I know this might sound very unrelated to anything right now just formulas on your screen but
there will be some additional re recommended reading at the end of this section so don't worry if you're
not picking up on the math.
Like even if we haven't explained the math right now.
But the point here is that what is across entropy well across entropy function.
Remember how we previously in artificial neural networks we had a function called the mean squared arrow
function which we used as the cost function for assessing our natural performance.
And our goal was to minimize the MSE in order to optimize our network performance.
Well that was our cost function then there and in convolutional neural networks we can still use MSE
but a better option in convolutional neural networks after you apply the soft max function turns out
to be the cross entropy function.
And in convolutional neural networks when you apply the cross entry functions not cost called the cost
function anymore is called the last function and they are very similar.
Theyre just a little terminological differences and like little bit different and on what they mean.
But for all purposes its pretty much the same thing.
And what happens is the last function is again something that we want to minimize in order to maximize
the performance of our network.
So lets have a look at a quick example on how of how this function can be applied.
So lets say we put an image of a dog into our network.
The predicted value for dog is 0.9 and this is doing the training so we know that we know the label
that is a dog.
So the predictive value 0.9 the prigged value for cat is 0.1 then here we have the label so we know
its a dog because this is training 0 1 for dogs or for cat.
And so in this case you need to use you need to plug these numbers into your formula for the cross entropy.
So how you do it is the values on the left going to the verbal cue.
The one that is under the logarithm in the on the right side and the values from the right would go
into P and so it's important to remember which one goes there because if you get them wrong you don't
want to be taking a logarithm for all me from zero value and or going from 1.
So you just want to plug them in.
Make sure you plug them into the correct places.
And then you basically add that up.
So that's how the cross entry works and we'll look at a actually right now we're going to look at a
specific step by step example of applying this function in real life and Ill kind of make make more
sense what Cross entropy is and it'll be less like that.
My goal in this toil is to make you more comfortable of cross century because it can sound very convoluted
and no pun intended it can.
Like convolutional neural networks it can sound very complex and scary but it's not.
That's that's the point.
So let's go ahead and apply it just so we know that it's not scary.
So here's your all that.
And also this will explain why we're doing this why we're looking into different cause functions.
So neural network one neural network let's say we have two neural networks and then we pass an image
of a dog and we know that this is a dog and not a cat.
And then we have another image our cat this time an animal and it's a cat not a dog and here we have
a we are looking at a hole which is in fact a dog not a cat.
If you look very closely.
So we want to see what our neural networks were will predict in the first case.
Neural network 1 90 percent dog 10 percent cat correct no network number to 60 percent dog 40 percent
cat still correct worse.
But correct.
Second option first neural network 10 percent cat dog 90 percent cat.
Correct.
You know that number to 30 percent dog 70 percent cat worse but still correct.
And then finally neural network in in image year old network won 40 percent dog 60 percent cat incorrect
neural network number to 10 percent dog and 90 percent cat incorrect and worse.
So the key here is that even though both net folks got it wrong in the last one through all three images
neural network one was outperforming neural network.
So even in the last case it was very it had it gave dog like a 40 percent chance as opposed to neural
network to only give dog a 10 percent chance or neural network one is outperforming across the board
when compared to neural network 2.
And so now we're going to look at the functions that they can measure performance that we've kind of
talked about the rating.
So let's put these into a table so there's neural network 1 you have the wrong number.
So that's the image number.
And then for image one you have.
What's it predicted 90 percent dog chimps and cat.
So there's the hat Marable's and then you have the actual value so dog correct cat incorrect.
Same thing for image number two and same thing for a minimum of three and same for neural network number
two.
So Dog 60 percent kept 40 percent in the first image.
That's what it predicted crotons was dog not a cat.
And so on.
And so now let's see what errors we can actually get.
So what errors we can calculate to estimate the performance and monitor the performance of our networks.
So one type of error is called the classification error.
And that is basically just asking it did you get it right or not.
Regardless of the probabilities is just DID YOU GET IT RIGHT.
Or did you get it right.
So in both cases for both neural networks each of them they got one.
So this is how you they go wrong.
So they got one out of three wrong.
So 33 percent error rate for your network 1 and 30 percent error rate for neural network.
As a baseline from this standpoint both neural networks perform at the same level but we know that's
not true.
We know that neural network Ikhwan is outperforming neural network.
That's why a classification error is not a good measure especially for the purposes of back propagation
mean square error different and by the way I did these calculations in Excel I just didn't want to bore
you with them but you can Tony just sit down and do them on a paper or in Excel.
These are very straightforward calculations just basically take the sum of squared errors and then just
take the average across your observations and that's pretty much it.
So for the for neural network one gets 25 percent for neural network 2 you get 71 percent error rates
so as you can see this one is more accurate.
It's telling us that nearly one has a much lower error rate than your own network.
And then cross entropy again.
We've seen the formula you can also calculate this is actually even easier to calculate than the mean
square error Cross area across entropy gives you 38 percent for neural network 1 and 1.0 6 for neural
network 2.
So you can see the results are a bit different.
When you look at them like that when you look at you know the miniskirt area and cross entropy and the
question of why would you use cross entropy over means squared error isn't just about the kind of like
the numbers that they say but all these calculations were just to show you that this is all it's all
doable you can just do it on a paper it's it's not.
It is not very intense mathematics.
These are pretty pretty simple straightforward things.
But the question of why would you use means cause entropy over means there is a very very good question
to ask.
I'm glad you asked that the answer to that is like there's several advantages of cross entropy over
mean squared error which are not obvious.
And so I'll I'll mention a couple but other then I'll I'll let you know where you can find out more.
So one of them is that if you for instance your at the very start of your back propagation your output
value is very very very very tiny very tiny.
So it's much smaller than the actual value that you want.
Then at the very start the gradient in your great and decent world will be very very low and you won't
be enough.
It be very hard for the neural network to actually start doing something and start moving around and
start adjusting those weights and start Movistar actually moving in the right direction.
Whereas when you use something like the cross entropy because it's got that logarithm in it it actually
helps the network assess even a small area like that and do something about it.
Here's how to think about it.
So let's say in again this is very in and in very intuitive approach.
There's going to be a link to the mathematics and you can derive these things through the mathematics
in more detail but a very intuitive approach.
Let's say your like your outcome that you want.
Is is one and right now you are at one one millionth of one.
Right.
$0.00 or is there one and then you improve next time you improve your outcome from from one millionth
to one thousandth.
And in terms of if you calculate the squared error you just subtracting one from the other.
Or basically in each case you're Kalka in a square and you'll see that the squared errors when you compare
one case versus other it didn't change that much.
You didn't improve your network that much when you looking at the mean square there.
But if you're looking at the cross entropy because you're taking a logarithm and then you're comparing
that to dividing one to the other.
You will see that you have actually improved your network significantly so that that jump from one million
to 1000 in mean squared error terms will be very low.
It will be insignificant and it won't.
It won't guide your gradient boosting process or your back propagation in the right direction.
It all it will guided in the right direction but it'll be like a very slow guidance it won't have enough
power whereas if you do recross entropy across entropy will understand that even though these are very
small adjustments that are just you know making a tiny change in absolute terms in relative terms it's
a huge improvement.
And we are definitely going in the right direction.
Let's keep going that way so cross entropy will help your neural network get to the right gets to the
optimal state is a better way for the neural network to get to get it to an optimal state.
But bear in mind that this only works when it across entropy is only the preferred method only for classification.
So if you're talking about things like regression like which we had in artificial neural networks then
you would rather go with me and squared error whereas cross entropy is better for classification and
again it has to do with the fact that we're using soft next function so that's a kind of intuitive explanation
of that a good place to learn a bit more about that if you're really interested in you know why are
we using cross versus mean square error.
Google a video by Geoffrey Hinton called the soft max output function and he explains it very well and
you know being the godfather of deep learning who can explain it better anyway.
And by the way any video by Geoffrey Hinton is golden.
He's just got a huge talent for explaining things anyway.
So that's that soft nice versus cross and I hope that gives you kind of like an intuitive understanding
of what's going on here.
But more importantly that you're not put off by the term cross entropy because headline will mention
it in the practical stories and I wanted to make sure that you're prepared for that.
And it's just another way of calculating your last function.
And another way of optimizing your network which is specifically tailored to classification problems
and therefore convolutional neural networks and comes in hand hand-in-hand with the soft max function.
So additional reading if you'd like a light introduction into cross entropy if you're interested in
the concentrate a bit more of course.
A good article to check out is called a friendly introduction to cross entropy loss by Rob DePietro
2016.
Here's the link below.
Very very nice very soft and nothing no super complex math.
Good analogies good examples using analogies of cars and you look at cars and talks about information
and bits and restrictions and you know how would you decode this whole Unico that it's so it's a good
article to have a look at and we'll give you a good overview of a cross entry like from an introductory
standpoint.
If you want to dig into the heavy math like what you see here then check out an article by or a blog
by how to implement a neural network Intermezzo too so in terms of use is like an intermediary thing
like a.
Intermittency in.
You know like when you go to a theater and you have like a break between the first part and the second
part.
So because he's like going through all these steps and then he's like and then he says I got to explain
this first.
And yes so that's why it's called intermezzo.
No other reason as far as I understand the articles by Peter Rolands 2016 as well so both are quite
recent.
And you know check out this if you'd like to dig into the mathematics behind Kross entropy behind the
soft Max and cross entropy in this article actually.
So there we go.
That's all there is to these two.
Hopefully I was able to add some additional clarity and good luck with that.
It's going to be fun and enjoy the practical tutorials.
I'll see you next time.
Until then enjoy learning.