‫Welcome back.
‫In the last video, we saw how we could use the predict method to make predictions with our machine
‫learning models, our fitted machine learning models on regression models or regression estimators in
‫the psychic loan terminology.
‫So let's check back in with what we're covering, what we're covering.
‫So we've done zero one, two, three.
‫We've made some predictions on our data because that's what the angle of the machine learning is, right,
‫is to make some kind of predictions.
‫We have some existing data.
‫We'd like to train an algorithm on that data to find patterns and use those patterns to make predictions
‫in the future.
‫So but just as important as making those predictions is evaluating those predictions.
‫So let's jump into that.
‫I'm going to write a new heading.
‫Here we go, 4.0.
‫Evaluating a machine learning model.
‫The reason we evaluate our models is because.
‫So we built a model to to predict something and then we wanted to put it in an application.
‫If our model is making poor predictions, would that be a great experience for anyone who uses that
‫application?
‫Probably not.
‫So before we put our models into the application or predict on something, that's going to be pretty
‫critical.
‫We want to evaluate the existing predictions that we can.
‫So with psychic learn, there are three ways to evaluate well, three inbuilt ways you can create your
‫own custom ways.
‫But three way is to evaluate psychic loan models, slash estimators.
‫These are part of the psychic loan library.
‫One is the estimators built in score method.
‫We've had a little bit of experience with this so far.
‫Two is the scoring parameter.
‫We haven't seen that one scoring parameter.
‫We may have actually we've covered a lot, but we're going to go through each of these and then problem
‫specific metric functions.
‫So depending on your working with classification or regression is going to be specific evaluation metrics
‫for each of those problem times.
‫And where do we get this from?
‫Well, this is coming in to the psychic loan documentation.
‫This is metrics.
‫And scoring is a very important piece.
‫This is your actual curriculum, by the way, to read through this, you don't have to know all of these
‫off by heart, but just understand where to find this information.
‫That's that's a big part of being a programmer and especially a machine learning engineer.
‫A data scientist is just knowing where to look, where to find out things.
‫So I'm going to put this here and I've taken the three different APIs.
‫So we have the estimated score method, the scoring parameter and the metric functions.
‫Let's start to break each of these down.
‫Hey, I'm going to bring that in here.
‫And then this is I'll put this here.
‫You can read more about these here.
‫Wonderful.
‫So let's just have a recap of the score method and I'm going to format this to make it look nice and
‫pretty.
‫Roll into that -- beautiful.
‫Let's have a recap.
‫So I'm going to put it a little another little heading here for point one.
‫We will go evaluating a model with the score method.
‫Turn that into mock down and we'll go from scale, learn ensemble, import, random forest classifier.
‫So we'll do it with our classification data first.
‫AMP Random Seed 42.
‫Now I want you I want you to give you I'll give you a challenge here.
‫Create X and Y.
‫I'm going to leave some comments here.
‫Create X and Y, create train test.
‫So this will be X equals heart disease, something we'll use our heart disease data set and then we
‫want to create.
‫I'll just go down here classifier model instance and then fit classifier to training data.
‫So that's my challenge to you.
‫I'm going to you can pause the video here and try to fill out these comments yourself.
‫But I'm going to return in three to.
‫One.
‫I'm back.
‫And so I hope you gave that a shot to fill out these.
‫But if not, that's okay.
‫We're going to go through them together.
‫We've written this code just up here.
‫Now, this is one of the only times I'll ever let you copy and paste code as if we've gotten stuff,
‫if we're rewriting stuff that we've written a number of times before.
‫So I'm going to copy this.
‫I'm going to bring this down.
‫We're checking out the score method.
‫That's the first one we're checking out.
‫So let's copy paste that.
‫So random seed, make the data, split the data, instantiate random forest classifier.
‫Now I believe, yeah, an estimates is 100 by default.
‫We'll just leave that there anyway.
‫Fit the model to the training data.
‫So let's let this run.
‫Beautiful.
‫So we've got a fit.
‫Random forest classifier, nice and fast.
‫And now how do you think we will use the score method?
‫Now, this is a bit of a trick question because we've we've just seen it above.
‫It actually came from the cell where we copied all of this.
‫So let's try it.
‫See?
‫Score.
‫And how about we try score it on their training data?
‫1.0.
‫Now, why might it get a score of 1.0 on the training data if we imagine?
‫The highest value for score, by the way, is what?
‫I'll just write this down explicitly.
‫The highest value for the score method is 1.0.
‫The lowest is 0.0.
‫So 1.0.
‫The default score method for a classification algorithm is accuracy return the mean accuracy on the
‫given test data and labels.
‫And now, of course, we're going to have to convert this to percentage if we want accuracy and percentage.
‫So 1.0 is 100% accuracy.
‫So if this was 0.8, it would be 80% accuracy.
‫Now, why is our machine learning model getting 100% a perfect score?
‫Have I told you yet to be skeptical of any machine learning model that gets a perfect score anywhere?
‫If not, consider that this is me telling you that always be skeptical of perfect metrics.
‫Why is our model getting 1.0 on the training data?
‫Well, that is because it's had exposure to all of the training features and all of the training labels.
‫And so if the model is powerful enough, it will achieve a perfect score on the training data because
‫it's able to split the data.
‫In our case, it's binary zero and one.
‫It's able to predict from all of the X train values.
‫It's able to predict perfectly all of the Y train values.
‫Now, it might be a different scenario for the test data.
‫Remember, it hasn't seen the test samples yet.
‫We want our model to be able to learn enough from the training features and the training labels to predict
‫on samples it hasn't seen before.
‫That's the crux of machine learning.
‫Learn patterns in data that we have existing to make predictions, quality predictions on data that
‫we haven't seen before.
‫Oh, so the score is a little bit less on the unseen data.
‫Now, this is a trend that you're going to see oftentimes in your own machine learning projects and
‫machine learning research papers everywhere.
‫The training score is usually sometimes quite a bit higher than the test score, but they should be
‫relatively close.
‫But the training will generally be higher than the testing score.
‫And if you ever get a perfect test score, like 100% accuracy or something like that, always be skeptical.
‫Go back and check your data.
‫But if we made our model a little bit worse, so maybe we only use five estimators.
‫Let's see what happens here.
‫A number of estimate as equals.
‫Five.
‫What's the training score?
‫You see how we get a less than perfect score?
‫How many can we push it to get?
‫Perfect.
‫There we go.
‫It only needs ten.
‫That's the power of an ensemble algorithm.
‫What if we just use two?
‫What does it get?
‫Still gets 90% with only two.
‫But how does that perform?
‫See, we get a much worse a score.
‫So that's the ensemble algorithm in a nutshell.
‫The Random Forest is one of the most powerful and fastest ensemble algorithms, so we use ten here.
‫We get a perfect score on the training data.
‫What if we pushed it up to maybe 500?
‫Can we push the test score higher?
‫We don't actually get that much of a higher test score.
‫Even if we use maybe 1000.
‫You see how you can tweak different things with machine learning models.
‫This is the fun part about it, right, is tweaking different things and getting different results.
‫Now let's push forward.
‫We've seen the score method for classifiers.
‫Remember the default score evaluation metric in classification models and psychic learn is what accuracy
‫you can find that out in the docstring.
‫Let's do the same.
‫Let's use the score method on our regression problem.
‫So
‫I'm going to leave that up to you.
‫I'd like you to replicate this, but for a regression data set and using a regression algorithm and
‫test out the score method, I'll give you a hint.
‫We have done this before, but I'd like you to try and write it out yourself this time.
‫Do the same setup, set up, make the data, split the data, make the model fit the model, score the
‫model.
‫I'll see you in the next video.