‫Welcome back.
‫In the last video we checked out mean absolute error and we saw that it calculates the average difference
‫or absolute difference between the actual values and the predicted values.
‫And I left off with the challenge of getting you to do the same with the mean squared error.
‫Now let's see that in action.
‫Mean squared error MSA.
‫Now you might have already looked into this and adhered to the challenge, but main squared error is
‫MSA is the main of the square of the errors between actual and predicted values.
‫Now I want you to have a think about what might this square do.
‫So if we get the differences here, if we square these values, of course they're all going to be positive.
‫But what might it do to values that are quite high?
‫Let's say, for example, that this was a value of we had an error of four.
‫This was 16.
‫What might that do?
‫Because it's going to be squared.
‫What's going to happen here if we had squared differences?
‫It's going to be huge, wouldn't it?
‫So just keep that in mind with main squared error.
‫When we square the differences, we're going to amplify values that are larger.
‫Whereas main absolute error keeps everything on the same scale.
‫Squared, of course, squares the error.
‫So let's have a look and say this in practice.
‫Mean squared error from sq.
‫Learned Metrics.
‫Import mean squared error.
‫And let's go.
‫Why?
‫Fred's equals model.
‫Predict.
‫We don't have to do this.
‫We've already got these instantiated but we're going to do it anyway.
‫Because what is evaluating a model comparing predictions to ground truth values MSI equals mean squared
‫error.
‫We're going to compare this to why test and why Preds.
‫Let's get the MSI here.
‫What do we get?
‫0.253.
‫Okay, so it's less than the main absolute error.
‫Now why might that be?
‫Well, let's, let's create a squared differences column.
‫I want you to have a think about why we do this.
‫So how can we do this.
‫We'll go diff differences that's going to be on our DF squared differences.
‫Let's make that squared differences equals np square DF differences and then let's have a look at df
‫dot head.
‫Okay.
‫So because we're squaring small numbers where I see they're getting smaller there.
‫All right.
‫Okay.
‫So that's just something you'll have to keep in mind what scale your values are in when you're squaring
‫them and when you're thinking about different differences is that if you want MSA to be in a certain
‫scale, think about the values you're putting into this error function or this evaluation function.
‫Now, let's again calculate this by hand.
‫So.
‫Calculate MSA by hand squared equals np square def and we'll get the differences.
‫Column differences squared don't mean.
‫What does that look like?
‫0.253 again.
‫Now this might be different if we were to change the scale of all of these values.
‫And I just want to emphasize how messy.
‫So the squared portion of it amplifies values or differences that are quite high.
‫So let's say we change this one to we'll give this.
‫So we had a difference of four and so the squared difference was 16.
‫So remember that these values are in hundreds of thousands.
‫So, so we had a difference.
‫Our model produced an ma difference of four.
‫So 400,000.
‫What does that look like?
‫When we calculate the MSA metric, we'll go def I log the first index.
‫So zero even though this is 20,046, we'll get that value and then we'll index into the squared differences
‫column and we'll change that to 16.
‫Oh, what have we got it now value is trying to be set on a copy of a slice of a dataframe.
‫That's alright.
‫We can just go on ahead.
‫We can ignore that warning for the time being.
‫They have done head def.
‫We want def.
‫Let's go.
‫We'll create a new def.
‫Actually def large era equals def copy and then def large error on lock.
‫Oops, we need to copy there.
‫And then we got def large error.
‫Beautiful.
‫So then we got a large error there.
‫It's a def large error.
‫We just made a copy of our original dataframe.
‫Now what do you think this is going to do if we calculate the MSSA on this value?
‫So this is the original one.
‫Let's get the average of this squared differences now.
‫So we want calculate mssa with large error.
‫Now of course this is we've manually fudged this data so manually put this in but it's just a demonstrate
‫what happens with MSSA.
‫So let's go def squared differences and we're going to get the mean of this column.
‫Oops, def large error because that's the one with the large error main.
‫What's happened to you?
‫Oh, wow.
‫It doesn't change much at all, even though we've introduced a large error.
‫So how about we create a few large errors?
‫Let's go def large error and we get I lock.
‫I see.
‫We want because we're still dealing with small numbers.
‫There's 20.
‫There's 4000 samples in here.
‫So the average works it out.
‫Let's go.
‫1 to 100 equals 20.
‫Just to really step this up.
‫And then df large era.
‫Getting a lot of typos here.
‫There we go, 20, 20, 20.
‫And now let's get the same with logic errors.
‫Yeah.
‫So we pump it up there.
‫Right.
‫That's with artificially inflated data.
‫Oh, and we've got differences here.
‫We've messed up we've messed up the the data frame here.
‫But that's okay.
‫This is just for example purposes.
‫In our case, the difference is our model actually performs pretty good.
‫So the differences aren't too large for any individual particular sample, which is which is a good
‫thing.
‫It means our model is doing quite well, but with MSSA it's going to penalize.
‫So if we were judging our model based on MSI and it had a difference here, that was much larger than
‫the rest and outlier, it's going to amplify that and really drive this number up because our model
‫actually did quite well across 4000 samples.
‫The squared error actually averages itself out to quite a good value lower than the NMA.
‫Now that isn't always the case because generally MSI because of the squared value is higher than the
‫MMA.
‫And as a partner, I just want to throw some more information at you, but not that you've had enough.
‫The main squared error also has an MSI, which is just the square root of the main squared error, but
‫that's going to be covered in this documentation here.
‫Metrics and scoring.
‫If we look up root mean squared error, you can find it there just taking the square root of the mean
‫squared error.
‫And finally, let's finish on which regression matrix should you use because we've seen a few now there's
‫a lot more, but these are three of the main ones you're going to come across.
‫So our squared coefficient of determination is similar to accuracy, gives you a quick indication of
‫how well your model might be doing, but it doesn't always tell you exactly how wrong your model is
‫in terms of how far off each prediction is.
‫If you've got an R squared value of one on your test data set, I'd be very skeptical.
‫So check that out.
‫Make sure you've done the data splits correctly.
‫MA gives a better indication of how far off each of your models predictions are on average.
‫So each prediction on average is this far away from the target value.
‫It's in the same scale as your target values.
‫And when you want to compare MI or MSI because of how MSI is calculated, it squares the differences
‫between predicted values and actual values.
‫So it amplifies larger differences.
‫So if you want to pay more attention to me when being $10,000 off is twice as bad as being $5,000 off.
‫So that's when the scale is correct.
‫But if your prediction is of 10,000, off is more than twice as bad as being 5000 off.
‫So if you need a model that needs to be really close to the value and the further it gets away, the
‫worse the error is.
‫You might want to try and reduce MAR, but keep these in mind.
‫Refer back to this slide.
‫Refer back to the code examples.
‫Refer back to the documentation of how to evaluate a model.
‫You'll come across this quite often.
‫As soon as you train a model, you'll want to evaluate it.
‫So with that being said, let's move on.
‫We're going to check out another way of evaluating models.
‫We're going to finally look at the scoring parameter.