‫Now we've tuned the hyper parameters of our logistic regression model and it's getting pretty good score
‫here.
‫But remember this score, the default score for a classifier is accuracy.
‫What we want to do now is create some evaluation metrics around our model that go a little bit beyond
‫accuracy.
‫More specifically, let's create our tuned machine learning model.
‫Actually, machine learning classifier is probably a better one because these evaluation metrics are
‫specific to classification, so we're going to go beyond accuracy.
‫So again, more specifically, we want an ROC curve and AUC score.
‫We also want a confusion matrix.
‫I'm putting fours in there that actually looked like it was from the Matrix with lots of numbers.
‫You know, that flowing green screen.
‫We want a classification report.
‫What else do we want?
‫We want precision recall and f one score.
‫And it would be great.
‫And it would be great if cross-validation was used where possible.
‫So let's turn that into markdown.
‫So this is what we're going to be working on.
‫We'll use our tuned grid search, logistic regression model as well as the best type of parameters for
‫it.
‫And we'll see where they come into play in a second.
‫So first of all, when we evaluate a model, it's always comparing how our trained model's predictions
‫compare to the truth labels.
‫So what we have to do is make some predictions first so we can compare them to the truth labels, a.k.a
‫the labels in the Y test dataset.
‫So let's do that.
‫Well, write a note here to make comparisons and evaluate our trained model.
‫First, we need to make predictions.
‫So what we'll do we're going to make predictions with tuned model and there's a beautiful function called
‫predict that we can use for that so we'll save it to Y spreads and we use GS log rig which is just our
‫trained version of our grid search model.
‫So GS log rig don't predict and we're going to predict on the test data why Preds Wonderful.
‫And let's see them just to make sure we're not going crazy, beautiful.
‫And now we need to compare them to the test data set.
‫So we have a look at this here.
‫Okay.
‫So if we wanted to go zero zero, we can know it's got this one wrong because that's supposed to be
‫a zero there because this is the truth labels here and then we could keep going through there, but
‫we're not going to do that.
‫We want to use some code to do that.
‫So the first thing first, we want a rock curve, because remember that we've got this little list up
‫here.
‫So a rock curve and AUC score.
‫Hmm.
‫What is a rock curve?
‫Well, if we look that up.
‫So what is a rock curve?
‫We went through this process.
‫So understanding AOC Roc curve.
‫So if we were to read through this.
‫What we would get is the rock curve is created by plotting the true positive rate against the false
‫positive rate at various threshold settings.
‫OC Beautiful.
‫So let's see how we do that.
‫The rock curve is a way of understanding how your model is performing by comparing the true positive
‫rate to the false positive rate.
‫And if we want to figure out what a true positive and a false positive is, we can have a look at our
‫confusion matrix for that.
‫So a true positive model predicts one when the truth is one, and a false positive is just the model
‫predicts one when the truth is supposed to be zero.
‫And a perfect model is going to get an AUC score.
‫What we see in a second of 1.0.
‫So let's see how we do it.
‫So import RC curve function from skyline dot matrix.
‫That's where it's from.
‫That's where it lives.
‫But we've actually already done this as with all the other line functions we've been using right back
‫up the top.
‫Right back up here.
‫So this is model evaluations right now in this next section, this next video, we're going to tackle
‫everything from here.
‫So these are we can see here we've got randomized search of grid search, key confusion matrix plot
‫curve that's we're about to use precision score, recall score, F1 score.
‫So these are for our classification model.
‫So let's go down.
‫We can save ourselves a line of code.
‫We don't actually have to import that.
‫We come right back down here.
‫We can just go.
‫Plot Roc curve.
‫Now, this is a relatively new addition to the psychic loan library, which I'm very, very happy with,
‫because usually you had to write these on your own.
‫This function, this plot curve and calculate it actually calculates the AUC metric for us.
‫So the area under the curve metric, let's see it in action.
‫I'm going to go plot ROC curve.
‫She's like Greg.
‫And if we look up this in the documentation, it'll tell us what it does, pass it an estimate, which
‫is our machine learning model Passat, Passat y plot receiver, operating characteristic, accurate
‫curve.
‫That's that's what that's going to do for us.
‫Beautiful.
‫So Jes log rig x test and we want it to do it on the test data set.
‫Always evaluate.
‫This should be calculate.
‫Always evaluate machine learning models in the test data set and let's see.
‫Oh, beautiful.
‫Okay, so remember a perfect rock curve we've seen in previous videos.
‫We go up to this corner and then across like that, but as Ted is up pretty well, then we get an area
‫under the curve.
‫So if we calculated all this area under here of 0.93, of course a perfect model will achieve 1.0.
‫So our model's not perfect, but it's an AUC score of 0.93 when the average of just.
‫Tossing a coin essentially would be 0.5.
‫So we're edging close to a perfect model.
‫Not bad for a model that just came out of the box.
‫All right.
‫The next thing we want to do is a confusion matrix.
‫So let's see that.
‫So confusion matrix.
‫How could we do that?
‫So I think it's like it Lone has a function and I don't think I know.
‫So like it alone has a function.
‫We can go confusion matrix and we want to compare the ground truth labels with the predicted labels.
‫Okay.
‫We could just leave it as that.
‫That's a bit bland.
‫We can improve the visualization of this confusion matrix using.
‫Seaborne.
‫That's what we want.
‫So we want to go.
‫I believe we've already have seaborne here.
‫Yeah.
‫Do we sneeze?
‫Yeah, we do.
‫So we've used Seabourn before in this project so we can go.
‫What I might do is I know ahead of time that we need a bigger font size.
‫So font size equals 1.5.
‫That's just sans dot set.
‫Now, we're going to create a little function here in case we wanted to make another confusion matrix.
‫Because psychic learns confusion, matrix function isn't up to scratch as is making this video.
‫You could always.
‫Maybe that's an opportunity for a pull request.
‫So we go here.
‫Nice and simple.
‫So we're just going to pass it.
‫This function, our test.
‫Labels and our predicted labels.
‫And then we're going to create a nice looking plot's, a nice looking confusion matrix using C bonds.
‫We've seen this before.
‫Heat map.
‫Wonderful.
‫So go figure.
‫X equals plt dot subplots fig size equals 330 shift into trigger happy again x equals sas heatmap.
‫Wonderful.
‫And we're going to pass it the psychic learn function sigh confusion matrix y test y pred.
‫And then we want to add here a not so annotate.
‫Yes, please.
‫Annotate equals true.
‫We're going to go C Bar because we don't want to Calabar because I've seen the Calabar and it doesn't
‫look too great on a confusion matrix PLT dot x label.
‫True label and then we're going to have plt y label going to go predicted label and let's see it.
‫So we're going to go plot comp format confusion matrix y, test y spreads.
‫Oh, no.
‫What have we done?
‫Font size.
‫That said, Alfonso is wrong.
‫What is this tab?
‫Oh, scale.
‫There we go.
‫Tab autocomplete for the went are classic and it's giving us this little cut off thing here so we can
‫fix this.
‫I believe by going bottom top it goes X dot get y lim Yep.
‫And then we want to go ax set, William.
‫Make it a little bit pretty set.
‫William And it's going to be bottom class, 0.5 top -0.5.
‫Now, of course, if your confusion matrix came out looking well, you might not need these lines,
‫but it doesn't matter.
‫There we go.
‫We might look a little bit better.
‫So you can see that the model gets confused.
‫So predicts the wrong label relatively the same across both classes.
‫So in essence, there's four occasions here where the model predicted zero.
‫The predicted label so predicted someone didn't have heart disease when they should have been predicted
‫as one.
‫So that's a false negative.
‫So it's predicting zero instead of one.
‫A false negative.
‫Remember, as we come back here, false negative model predicts zero when the truth is one.
‫And over here is the false positives.
‫So a.k.a, we've got three instances here where a model predicts one, someone does have heart disease
‫when they actually don't.
‫And you can see these are both things that we want to avoid.
‫Right.
‫Especially when we're thinking about something as severe or as serious as heart disease, even predicting
‫when it's not present.
‫So here, if we predict zero.
‫So no disease when it is present.
‫Okay, that's bad.
‫But also predicting that it is there when it's not actually there, that's also bad.
‫So that's something that you want to have to consider when you're building these type of models.
‫Is a false negative worse or is a false positive worse?
‫And again, a perfect model would have none of these, but in reality, you're going to probably end
‫up having some sort of confusion here with your model.
‫You're not going to ideally, these would be perfect across here, across the line.
‫Now we've done a confusion matrix.
‫We've got that there.
‫We could share that with our boss.
‫A ROC curve there.
‫What's next?
‫Classification report.
‫Okay.
‫And it would be great if cross validation was used where possible.
‫All right.
‫So that's what we might tackling the next video.