‫In the last video we left off saying that these metrics here, precision recall and F1 score are only
‫calculated using one split train test split so only calculated using one, but we want to prefer using
‫cross validation where possible.
‫So that's what we're going to do.
‫Let's make a little heading.
‫We'll go.
‫Calculate evaluation metrics using cross validation.
‫Well, right here we're going.
‫To calculate precision recall and f one score of our model using cross validation.
‫And to do so, we'll be using me.
‫Put this one in code cross vowel score.
‫So we've seen this one in the in the socket learn section, cross vowel score.
‫And if we look it up here as we always want to do so cross vowel score.
‫What's it say?
‫So it takes an estimate, it takes an X, it takes a Y, evaluate a score by cross-validation.
‫The thing we want to pay attention to here is scoring.
‫So if we go to scoring string, call a string or score our callable object function with signature,
‫which should return only a single value.
‫Okay, so this is the parameter we're going to use to evaluate our model using cross validation with
‫different metrics.
‫So we're going to change the scoring string.
‫So let's see that we'll set up a new logistic regression model instance with our best type of parameters.
‫What were our best type of parameters?
‫So let's go check best hyper parameters because we want to set up one of the best models that we can
‫and then use cross validation on it to evaluate it.
‫Jess Log rig dot best params.
‫Wonderful.
‫So now we're going to create a classifier, go here, create a new classifier with best parameters that
‫we found.
‫So CLF equals logistic.
‫Regression see equals.
‫Are you going to just copy that here?
‫Wonderful.
‫Oops.
‫I forgot the zero.
‫That's a very long number there.
‫And then solver, he calls.
‫Now, again, these parameters might be different if you've done your own high parameter tuning and
‫found some better ones.
‫But that'll do for us for now.
‫Wonderful.
‫So now we've instantiated a model with the best high parameters.
‫Let's use cross vowel score along with the scoring parameter to get some cross validated metrics.
‫So what we do, I've actually forgotten.
‫Go here.
‫We're going to calculate.
‫Yes, there we go.
‫So we want accuracy.
‫Go here.
‫Cross validated accuracy.
‫So we might do four cells and we go here cross validated precision.
‫Cross validated recall.
‫And then cross validated.
‫If one.
‫Scott Wonderful.
‫So let's do that.
‫So we're going to CVAC, which stands for Cross-Validation Accuracy.
‫Equals cross vowel score and we're going to pass it out classifier, which we just instantiated here.
‫We're going to pass it all of the X data because we can now, because we're using cross vowel score
‫and all of the Y data, we're going to set CV to five and we're going to set scoring to be accuracy.
‫Beautiful.
‫Then we're going to check out what CB Ark looks like and we can take the main of this because remember,
‫what has it done?
‫It's evaluated our model over five different splits.
‫So if we take the mean of it.
‫Right.
‫That's going to get the average accuracy across these five different splits.
‫So let's do that.
‫So we go MP dot main save a.
‫Wonderful.
‫So that's how accuracy there.
‫So we might save that.
‫Actually, you might just override our value and then we'll go save a.
‫Beautiful.
‫And now let's do the same with these other ones.
‫We'll go here.
‫We could just copy the code here.
‫Again, you don't want to get into the habit of copying code.
‫We should really function as this, but we're just going to keep rolling with the punches.
‫What we're doing now and by function is this I mean, because we're doing relatively similar calculations
‫the whole way through, we could just make it a function so there's a precision might override this
‫to be precision equals MP mean.
‫And then see it again.
‫Wonderful.
‫And then we can do the same thing here, except for recall.
‫What we might do is be a little bit tricky to put in spaces here when you hold command.
‫You can put little curses here.
‫So if we just change this all to recall.
‫Ha!
‫Happy days.
‫Look at that.
‫I got five curses on the go recall.
‫Wonderful.
‫Oh, that's a nice recall score.
‫Cross validated as well.
‫We might do the same for EF one.
‫We go here, we're going to change all of these bad boys to f one.
‫There we go.
‫We might put a little spice here, so our code is looking polyphonic.
‫Wonderful.
‫All right.
‫And so now we've got all these cross-validation metrics or cross validated metrics that our boss is
‫requesting.
‫It's not really good to just have him here sitting in a Jupyter notebook like all of our other valuable
‫staffers in a nice, neat little table.
‫There's our classification matrix that looks really good in a presentation.
‫You know, this is something that someone could look at and go, Yeah, I can see that value is pretty
‫high.
‫I can see what's happening here.
‫Let's do the same.
‫Rather than just having them all spread out.
‫Let's put it into a graph of sorts or a visualization.
‫So we go visualize our cross validated metrics.
‫So we're going to create Save a Matrix as a data frame.
‫I mean, yeah, we probably really should have functionalized this to begin with, but that's right.
‫Sometimes you have to just go through the.
‫Old fashioned way and then realize where you've gone wrong and see how you could improve it.
‫Maybe that's a little extension.
‫You could try yourself and see how you could function.
‫Some of what we're doing here, maybe it just cross validates X and Y using different metrics and puts
‫it all into a nice little presentation all in one hit.
‫That would be a good practice.
‫Save a recall.
‫And then we go f one is going to be our CV f one.
‫Wonderful.
‫We need to set an index equals zero.
‫View of, Oh, then we can create a plot, save a matrix.
‫Now I've done this before.
‫Spoiler alert, so I know I need to transpose it.
‫Bar title equals cross validated classification matrix.
‫And do we want a legend new we don't false.
‫Let's see what that looks like.
‫Beautiful.
‫We're going to put a little semicolon here.
‫There we go.
‫So maybe we could probably add, like, the numbers on here so people know exactly what they are.
‫But we see our model's doing pretty well on recall.
‫And again, we could probably add our cross-validation results from our random forest model or our K
‫nearest neighbors classifier or any other classification model that we may have tried during our experimentation
‫phase.
‫Remember, this experimentation phase is just it's an iterative process, right?
‫Just going back and back and back and forth through here.
‫So we could add that to here and see which model performs best.
‫But for now we're sticking with logistic regression and this looks like something that we could share.
‫So what do we have left now?
‫Well, let's scroll back out to where we put down what we've what we've ticked off.
‫Have we ticked off everything?
‫The things our boss was asking, the things we previously didn't know how to do.
‫But now we've seen them in action.
‫Whoa.
‫All right.
‫So we've got.
‫Cross-Validation Precision Recall yep.
‫Classification Report tic rc curve tic area under the curve confusion matrix.
‫Yep.
‫The thing we're missing out on here is feature importance.
‫Hmm.
‫All right, well, let's see how we do feature importance in the next video.