‫Welcome back.
‫Now let's have a look at the third way of evaluating cyclone models.
‫And if we jump back into the documentation, that's through metric functions, specifically in scale
‫and metrics.
‫And we have classification metrics which we see accuracy score.
‫Have we got precision?
‫We've seen a few of these actually before, which is a classification report confusion matrix.
‫There we go, precision score, recall score.
‫And then we have the same again for regression metrics.
‫We've got a few here.
‫Mean absolute error, mean squared error.
‫I issued the challenge in the last video.
‫Oh there's R2 score.
‫Hello.
‫There was to try out a few of these on the problems we've been working on so far.
‫I hope you gave that a go because that's the best way to start practising on your own, is to just explore
‫the documentation, write some code, make some errors, and then just see how to fix them as you go.
‫Now let's write down a little heading using different evaluation metrics as psychic loan functions,
‫and we'll go here.
‫The third way to evaluate cyclone machine learning models slash estimate as they use the estimated term.
‫I like the term model is to use the cyclone dot matrix module and then we'll just link the documentation
‫there and turn that into markdown.
‫Beautiful.
‫Let's get into start writing some code.
‫We'll start with our classification problem.
‫We'll put from cyclone matrix, import accuracy score.
‫We'll get the accuracy, we'll get the precision score and we'll get the recall score and maybe, maybe
‫that'll be enough.
‫Oh, how about we do f one as well?
‫Why not?
‫We'll have a little bit of completeness there and then we'll get a model.
‫So from Cyclone Ensemble, we want to import the random forest classifier and then what else should
‫we get?
‫Well, we might as well get trained to split.
‫Why don't we do that?
‫From SRC Learn model selection.
‫We'll go through the whole workflow again.
‫Selection, import, train, test, split.
‫And so I want you to give this a go.
‫Create X and Y.
‫Then I'd like you to split the data, then create model Y, fit model, and then evaluate model using
‫evaluation functions.
‫So the ones we just imported here, so you'll have some predictions and checking out the docstring to
‫these if you want to figure out what they are.
‫These are just the main classification metrics that we've covered accuracy, precision, recall and
‫F one.
‫There are a few more here, of course, for classification.
‫If you'd like to look at any more of them, please jump into the documentation.
‫I'm just sharing some of the ones you'll see most often.
‫Same from regression.
‫There's a fair few more here than what we've covered.
‫If you'd like to read more, the documentation is here to go.
‫Now.
‫I'll let you give that a shot to fill that out.
‫In the meantime, how about we get ready to do one for regression as well from scale metrics import
‫R to score, which is a coefficient of determination.
‫Main absolute error.
‫The main value of how different your model's predictions are to the actual values mean squared error
‫the squared value of your model's predictions or the error of your model's predictions compared to the
‫actual labels.
‫And then we're going to get skylon.
‫We'll get the ensemble module, and we will import our faithful random forest regression.
‫And we'll also get we'll get the same thing that we have above.
‫Don't always have to write this because it's already imported.
‫But just so each of these pieces of code is stand alone, I'm going to copy this down in here.
‫It's going to be the exact same sort of layout.
‫That's the beauty of the psychic loan library, is that much of the workflow is similar depending on
‫what problem you're working on.
‫So you can pause the video here, try to fill these both out.
‫If not, we're going to continue in three or so seconds, 3 to 1.
‫I hope you gave it a shot, but let's let's now do them together.
‫Hey, so we need to create X and Y.
‫We might start off.
‫This is one thing I missed out here, actually.
‫Random seed and we'll go x equals heart disease.
‫We've done this a few times now, but we'll drop the target on the first axis and then we'll set Y equal
‫to the target value heart disease target split data.
‫So if we can do this off by heart x train x test comes first.
‫Took me a while to learn that order, but just remember x values first train test then y values train
‫test.
‫Train test split.
‫We're going to split x and y.
‫Remember, this is only going to split it for one split rather than what we saw before, which was cross
‫validation.
‫Splitting it into end splits or K splits is a term that it's often used.
‫K could be three K could be two K could be ten K could be five.
‫Passing the random forest classifier.
‫We'll leave it at the default parameters.
‫We'll fit the model on the training data.
‫Self don't fit on x train y train features and labels.
‫Now we're going to evaluate the model using evaluation functions.
‫However, what you might have found is another thing that I've missed out, so maybe I've intentionally
‫done this classification matrix on the test set is how to specifically use these functions.
‫Now, accuracy we've seen, we may have seen them before, but that's okay.
‫If we haven't, we're going to cover them.
‫We've we've done many similar steps to what we're about to see.
‫We have the Y test values.
‫But if we look into the documentation of accuracy score, what do we get?
‫Accuracy, classification score.
‫We need to compare the Y true values, which in our case are the test labels to the Y prediction values,
‫which are our model's predictions.
‫So we could put in here self predict X test, but rather than rewriting self predict all the time.
‫So if we just go there and this should give us a value times 100 and we'll format it to two decimal
‫places, put a little percentage sign at the back here and see what this comes out.
‫X Yes does not define.
‫Do we need that as x test?
‫There we go.
‫Okay.
‫However, why don't we just make some predictions and we just reuse that?
‫So make predictions we want self.
‫Predict on s x test and the Y spreads are going to be equal to that.
‫So if we just copy this line, we should get the same value for y spreads because it's just the same
‫thing except we're just putting it in a variable.
‫Yeah, there we go.
‫So that's a bit better.
‫And we're going to get rid of this, delete that.
‫And how about we take this, copy this because we want one for precision recall f one.
‫Which are these metrics here?
‫Precision recall F one once again, we could use a few more if we wanted to because we've got a bunch
‫of classification metrics here.
‫But these are some of the most common, some of the most important.
‫We're going to change this to precision.
‫We'll change this one to recall, and we will change this one to F one.
‫So once again, we can do the same thing.
‫Precision, precision school, recall score and f one score.
‫And then we'll just we don't need to format them as percentages either.
‫That will do.
‫Accuracy is the one you'll find often as a percentage.
‫Oh, what did we miss here?
‫Oh, we missed the end of the docstring or the formatted string.
‫Did you pick up on that?
‫There we go.
‫Beautiful.
‫We could cut them down to a couple of decimal points, but that's all right.
‫So there we go.
‫We've now seen the third way of using a valuation metrics as second line functions.
‫Now this is for classification.
‫I'm going to pause the video or let you pause the video again and see if you can replicate what we've
‫done for classification.
‫You will have to add in an extra step here, making some predictions with the models, because what
‫is evaluating a model?
‫Evaluating a model is comparing our model's predictions to what the ground truth data is very important.
‫As important as training a model is evaluating that model.
‫So give that a shot.
‫Otherwise, we're going to continue and write it together in 3 to 1.
‫Welcome back.
‫I hope you gave that a shot of evaluating a regression model using some evaluation functions here.
‫And we've got all of these from the metrics module, the regression metrics module, from the cyclone
‫documentation.
‫And if you're wondering why is Daniel referring back to the documentation so much?
‫Well, it's because you're going to be doing that a lot, and I in fact, do that a lot.
‫So I'm trying to impart on you some of the habits that I use day to day as a machine learning engineer
‫or data scientist.
‫We'll create X and Y here, and referring to documentation is one of the most important habits you'll
‫ever develop.
‫Of course you can search things, but eventually you're just going to revert back to reading the docs
‫or searching, reading the docs, searching, reading the docs, writing code, getting it wrong, getting
‫it right over and over and over again.
‫Experiment, experiment, experiment.
‫So we'll create the split data.
‫We've seen that before.
‫We're going to instantiate the model as a what?
‫Because we're working on a regression problem.
‫We'll set it as a random forest aggressor.
‫We'll leave it with the default parameters we're going to fit the model on the training data model don't
‫fit X train, y train.
‫And now we can make predictions.
‫That was the extra step we had to add in here, which is just model predict on X test.
‫We'll set this up as a variable Y spreads, we could rename it.
‫So it's not the same as what's up here, but that's alright.
‫And now we're going to evaluate using regression matrix on the test set so we can go print F, let's
‫get the R two score first.
‫So we want R to score.
‫We're going to format that with R to score with y test y pred and then we want, oh we haven't got the
‫end of the string there.
‫We want print.
‫What else did we have mean.
‫Absolute error.
‫So ma can be mean.
‫Absolute.
‫Do we have?
‫Yes, we have auto complete y pred.
‫And then finally print we'll get what messy.
‫Messy is mean squared era.
‫Why test why pred and then finish that off there.
‫Oh no.
‫What did we get wrong?
‫I just kind of learned matrix mean squared error.
‫We got a typo.
‫Did you catch that before.
‫Mean squared error.
‫We run that.
‫It's going to fit.
‫A model might take a little bit longer than the classification one because our housing dataset has 20,000
‫rows.
‫Oh, what do we get wrong?
‫Y pred is not defined.
‫Oh we need y preds.
‫There we go.
‫He might have caught that typo again.
‫Just retrain the model there.
‫Reevaluate it.
‫So again, what's the difference between our cross val evaluation and our regression metrics or classification
‫metrics here?
‫These metrics are found on one train test split versus these metrics up here are found on multiple train
‫test splits using cross validation.
‫These metrics take longer to find because you have to train multiple models and evaluate them.
‫However, they are generally a better indication of how your model will perform outside of a Jupyter
‫notebook on unseen data, then just on one single split.
‫So just keep that in mind.
‫Beautiful.
‫We have some regression metrics on the test set, so we have covered it's taken us a while, but I wanted
‫to spend a lot of time on this section because it is so important to evaluate a model.
‫So if you want to read more on this, please refer to the documentation.
‫We've covered the estimated score method, we've covered the scoring parameter and we've looked at metric
‫functions.
‫There's a lot more here, but you've seen the crux of all three of these.
‫So now let's figure out what we're covering.
‫We'll have a look at our list.
‫What have we gone through so far?
‫Zero and end to end workflow.
‫Getting the data ready.
‫Choose the right estimator slash algorithm for our problems.
‫Fit the model.
‫Evaluating a model and oh number five, we're up to improving a model.
‫This is very, very exciting.
‫So we've covered a lot in the Psych Learn section so far, but we've still got a little bit more to
‫go.
‫I'll see you in the next section.
‫Let's figure out if we've got these metrics.
‫How might we go about improving them, making them better?
‫We've we've hinted at it in previous videos, but now let's really dive into it.
‫I'll see you in the next section.