‫Okay.
‫Welcome back.
‫Let's now get into some regression model evaluation metrics.
‫So make a new heading here.
‫4.2.2 regression model evaluation metrics.
‫And once again, we can go into the model evaluation metrics, documentation and see what's in there.
‫So if we go up to here, we've got model evaluation.
‫We've got a whole bunch of different evaluation metrics here.
‫Do we have does it actually explicitly say regression, log loss, binary case regression metrics?
‫Here we go.
‫Beautiful.
‫So this is your extracurricular for.
‫This video is to read through regression metrics.
‫We have some of the some of those have been enhanced to handle the multi output case means good error
‫just to read through this max error mean absolute error.
‫These are all different types of regression metrics that you can use.
‫We're going to focus on a few of the main ones, such as mean absolute error.
‫You'll often come across main squared error is the main squared log error.
‫Okay.
‫Yeah.
‫So we've got a fair few options there.
‫There are two score.
‫This is the default as we've seen before in the score method.
‫So if you'd like to see an explanation of all of these, please go through the documentation.
‫I'm going to encourage you to at least read it for the values that we're going through, because, again,
‫evaluating a machine learning model is just as important as training one and making predictions with
‫it.
‫But let's get hands on.
‫We'll copy this and we'll just click here for the heading, and I'm going to copy that, put it in there.
‫And so let's turn this into markdown.
‫Where are we at?
‫There we go.
‫The ones we're going to cover are number one can be squared.
‫Now, this is that's a symbol for it, but it's pronounced as you've heard me before.
‫Pronounced.
‫Have I spelled that right?
‫It doesn't matter.
‫You know what I'm trying to spell?
‫Pronounced R-squared or coefficient of determination.
‫We're going to focus on that 1/1 coefficient.
‫There's probably some spelling mistakes there, but we're not focused on spelling mistakes for now.
‫If we had code errors, we'd fix those spelling mistakes.
‫You can fix those if you like, and then mean squared error.
‫Focus on those.
‫But if you'd like to, your extension is to read this documentation and practice on another one.
‫The beauty of a psychic loan is that they all use similar interfaces.
‫And what's the crux of evaluation?
‫It's comparing a model's predictions to what it should have predicted.
‫So let's get our evaluation or regression model back in town and we'll make some predictions with them.
‫So from Scalar Ensemble, we've done this a few times now we'll get the random forest regression, we'll
‫set our random seed and we'll set up our data set, which is x equals housing.
‫DF Housing DF dot drop will remove the target because we want to use the features X to predict the target
‫y housing DF and the Y is the target.
‫We'll create our train test split.
‫This is just code we've written before.
‫So if there is code we've written before, you might want to copy and paste it.
‫However, it is good to get a little bit of practice, especially if this is a lot of this is new to
‫you.
‫Split the data into train and test set y so we can train our model on the training data and evaluated
‫on the test data data our model has never seen before.
‫So we're going to set up our model random forest aggressor.
‫And because of the nature of the ensemble algorithm, we'll set an estimators.
‫So the number of mini models in our random forest, the number of decision trees in our random forest
‫to 100, which is the default.
‫Remember, Ask, Learn has a great set of defaults for many of its inbuilt machine learning models and
‫algorithms.
‫Just because it's so widely used, a lot of the default parameters will get you pretty good results
‫from the get go.
‫If they don't, you can always tweak them.
‫So let's run.
‫This shouldn't take too long and then we'll go model score.
‫We're going to score it on the test data set.
‫Now we've we've seen this before, we've written the exact same code before.
‫We get the coefficient of determination because it's a regression algorithm and then we've got evaluation
‫metrics here.
‫We've got a whole bunch here.
‫We've looked it up.
‫What it is, if you want to read more, you can read more.
‫But it's the proportion of the variation in the dependent variable that is predictable from the independent
‫variable.
‫Now how much of the target variable in our housing data set can be predicted by the feature variables?
‫The dependent variable?
‫How much can be predicted of this by the independent variables?
‫Now with the score, the highest value is 1.0.
‫I believe I might have said that the lowest value is 0.0.
‫Now that's for accuracy.
‫You can actually get a negative R2 score, can go to, I believe it's like negative infinity or something.
‫But I also said that if you predicted just the mean, so if you have all the Y test labels, let's have
‫a look at those Y test.
‫And then if we look at y test mean.
‫So if our predictions, if our model just predicted 2.055, etcetera, etcetera, for all of these values,
‫we would get an R two score of 0.0 and if it predicted the Y test perfectly, it would get a score of
‫1.0, the highest.
‫So very similar to accuracy in that case, a model that predicts very poorly gets 0% accuracy and classification.
‫If it predicts everything correct, it gets a score of 1.0 100% accuracy.
‫But as always, if you ever get perfect evaluation metrics, always be skeptical.
‫But let's see this in action.
‫I've said this out loud.
‫Why don't we see it working so from scalar and matrix import R2 score?
‫And this is another way that we can evaluate our models using psychic learn and we're going to see this
‫in a little bit, but that is right back up at the top when we scroll back to the top of this page.
‫That is a metric function.
‫So each metric in psychic loan has its own specific function regression metrics.
‫There we go.
‫That's where we're at R to score.
‫This is what we're using so we can import that.
‫And now let's fill an array with the y test mean just to demonstrate us getting a score are two score
‫of zero and P for this just means not only create a full array with of length y test with the value
‫y test mean that's all it's doing.
‫NumPy create a full array of length y test so the exact same number of values that are in the test data
‫set 4128 each with the value each.
‫So each sample is going to have the value of the test mean.
‫And then how might we check this.
‫Well, let's check that and then y test mean.
‫Let's look at the first ten.
‫It should all be what value?
‫It should all be this value.
‫So let's let's make sure that it is.
‫There we go.
‫Okay, beautiful.
‫Now, how might we use the AR to score function?
‫Well, if you weren't sure, how could you find out to score shift tab docstring.
‫Yeah that's what I'm talking about.
‫Look at that.
‫So we have r to score y through y pred.
‫So what are the Y true values in this case?
‫Well, we've said that if our model was just to predict the mean, which is all these values, then
‫we should get an R2 score of zero.
‫So these are our y pred so y true is going to be equal to y test and then we're going to have y pred
‫is equal to y test mean.
‫Let's see this.
‫I've said that if our model predicted just the mean, we'd get an R2 of zero coefficient of determination
‫of zero.
‫There we go.
‫So that's pretty cool, right?
‫So, you know, if your model is performing pretty poorly, it's going to get another to score of zero
‫or it could go negative.
‫But if your model is, what's the simplest model that we could use on our test data?
‫Or even on our training data as well.
‫The simplest model would be that we just predict the target as the average of all the other targets.
‫So that's a pretty simple machine learning model, right?
‫But the power of the random forest is that it's able to achieve a model that gets R to score of 0.8
‫close to 0.81.
‫A perfect model would get 1.0.
‫Let's have a log of that to score y true equals y test and then y pred would equal the same thing.
‫If our model just perfectly predicted the test data, it'll get an HA to score of 1.0.
‫So that's a quick way that you can test your regression algorithm to see how well it's doing at predicting
‫the target value from the features.
‫Now let's dive into another metric I've said we're going to use mean absolute error.
‫So I'd like to throw this over to you.
‫This is your challenge before the next video.
‫Actually, you can try both of these.
‫So try both of these mean absolute and mean squared error.
‫The setup is going to be very similar to what we've just done.
‫You'll have to make some predictions with the model and then you'll have to compare them to something.
‫So give that a shot.
‫If you want the documentation, you've got it here.
‫Mean squared error, mean absolute error.
‫And I'll see you in the next video.
‫We'll do it together.