‫Now we've just covered accuracy as a method of evaluating our classification models, and you might
‫be thinking, why can't we just leave it at that?
‫Well, as we start to go through these other metrics here, you'll start to understand why it might
‫be important to get a few more different evaluation metrics on the board rather than just accuracy.
‫In my case, when I was first learning how to build classification models, I would always think, Yeah,
‫just higher accuracy is better, right?
‫And then slowly, after looking at different examples, I start to realize, okay, now I see the value
‫of using other metrics like these, but let's not talk about it.
‫Let's see it in action.
‫The next one we're going to cover is the area under the receiver operating, so area under the receiver
‫operating characteristic curve.
‫The beautiful thing is it's also known as AUC, so area under curve or rock so receiver operating characteristic.
‫So let's do this rock curve.
‫So these are the two things you'll look out for area under curve or rock curve maybe we put it here
‫are you see or rock curve beautiful.
‫So if you hear someone talking about aiyuk or rock or rock or something like that, they're probably
‫talking about this metric here.
‫Now, what does a rock curve measure?
‫Well, by formal definition, a rock curve is a comparison of a model's true positive rate, a.k.a TPR
‫versus a model's false positive rate.
‫Let's write that down.
‫Here are a comparison of a model's true positive rate, which is also known as TPR versus a model's
‫false positive.
‫Right, which is also known as FPI.
‫Now, you might be wondering, okay, what is true positive and what is a false positive?
‫Well, let's have a look here.
‫A true positive Eagles model predicts one when truth is one.
‫Does that make sense?
‫So in our case, if we have a look up here, if we our targets are one or zero, a true positive is
‫when our model predicts a one.
‫The real label is a one.
‫Okay.
‫Yeah, that makes sense.
‫So if we go here, a false positive is model predicts one when truth is zero.
‫So why is it called false positive?
‫Well, it's because it's predicting one.
‫So.
‫Okay, the positive class has heart disease when the truth is actually zero.
‫So it's giving us a false positive sense that that a person may have heart disease in in our particular
‫case, for our heart disease classification problem.
‫And then if we go here a true negative.
‫He goes model predicts zero when truth is zero.
‫So that makes sense.
‫So that's predicting the model is getting the right prediction there.
‫It's predicting someone doesn't have heart disease when they actually don't.
‫That's great.
‫And then a false negative equals model predicts.
‫What do you think this one will be?
‫False.
‫Negative.
‫We look at false positive.
‫A false negative is when model predicts zero when truth is one.
‫So that's because it's predicting not heart disease when it's actually is heart disease.
‫Right.
‫So now we know these we can see that a rock curve is a comparison of a model's true positive rate.
‫TPR versus typos everywhere.
‫Here Daniel.
‫Come on.
‫Versus a model's false positive rate or fpr.
‫So now we know this.
‫Let's see it in action so we can do this using psychic loans, metrics library.
‫We're going to import rock curve.
‫So see how we're saying rock curve or I say curve and then we're going to make predictions with probabilities
‫and how can we make predictions with probabilities?
‫We saw this in a making predictions.
‫We can do that with y probs equals clf predict prober for probability.
‫We're going to make some predictions on the x test data.
‫Now I just want to make sure maybe we do that.
‫We create create x test just to make sure it's the right x test data x test.
‫ET cetera.
‫So we want to go here.
‫X train, x test.
‫Why train?
‫Why test equals train test split x y test size equals 0.2.
‫Wonderful.
‫And so now we can do that there because a rock curve is a comparison of a model's true positive rate
‫versus a model's false positive rate.
‫We want to only keep the positive classes.
‫So actually let's see what y probs looks like.
‫Why prompts?
‫Why hasn't this worked?
‫What do we got here?
‫This random forest classified is not fitted yet called fit.
‫Oh, that would make sense.
‫So we want.
‫Fit the classifier.
‫Of course, you can't make predictions without without fitting them, without the model, learning any
‫patterns.
‫So we're going to fit it here.
‫Then we'll make some predictions and we'll have a look at this.
‫Maybe only the first ten.
‫So we're not taking up heaps of space.
‫Okay, beautiful.
‫And so because our rock curve.
‫Only is a comparison of a model's true positive rate versus a false positive rate.
‫We only want probabilities that the model has predicted for the positive class.
‫So if you imagine here our models trying to predict zero or one.
‫This is the probability that the label is zero.
‫And this is the probability that the label is one and so on for for all of these samples until, until
‫it finishes up.
‫So there's going to be however many are in the test set here.
‫61, so is going to be 61 of these.
‫So essentially what we want is y probs positive.
‫So this is the probabilities that it's the positive class act.
‫Zero is the negative class and one is a positive class Y.
‫So we're going to use some slicing here, but only column one of every row so that we can do that.
‫Just so you can you know what's going on.
‫Positive.
‫And we'll only look at the first ten again.
‫So does this make sense?
‫We're getting 0.43.
‫Yeah.
‫0.77.
‫Yep, 0.48.
‫So on and so on and so on for the entire list of y probs.
‫And now we can calculate pr, tpr and thresholds.
‫So we go fpr it goes tpr thresholds.
‫Why do I know this?
‫We'll see this in a second.
‫Why do I know that this is going to be FBR comment EPR comma thresholds.
‫We've got ROC Curve, we're going to pass it y test and y probs positive.
‫Now, how would you figure out what rock clove returns in this case?
‫Well, what I would do is I would press shift tab and this is how I know.
‫So we got roc curve.
‫It takes y.
‫True.
‫So that's our test labels.
‫It takes Y score.
‫So that's our y probs positive.
‫And if we come down here, compute the receiver operating characteristic rock.
‫Yep.
‫Beautiful.
‫Now parameters.
‫Why?
‫True.
‫Why?
‫Score.
‫Target scores can be either probability estimates of the positive class confidence values or non threshold
‫measures of the decision.
‫So that's the probability estimates of the positive class.
‫That's where we got this slice from.
‫Right.
‫And then if we come down here, it's going to tell us what our returns fpr increasing false positive
‫rates such that element ie is a false positive rate of predictions with score above thresholds.
‫TPR So that's a true positive rate and thresholds.
‫So that's how we can figure out what a function returns.
‫And again, that's just viewing the docstring in a Jupyter Notebook.
‫You can always look at the documentation for this.
‫If you wanted to do so, you might look up something like How to calculate rock curve for Psychic Learn.
‫So then we're going to check check the false positive, right?
‫Appear beautiful.
‫So that's giving us a big array.
‫But looking at these on its own doesn't really make much sense.
‫It's much easier to see it visually.
‫And since Cyclone doesn't really have a built in function to plot a rock curve, what we might have
‫to do is and you're quite often you come across this right, is you'll have to find a function or write
‫your own that'll do it for you.
‫So that's what we're going to have a look at in the next video.
‫We'll plot a rock curve and this function here, rock curve, will start to make a bit more sense rather
‫than just being an array of numbers.