‫So the final classification model evaluation metric we're going to have a look at is a classification
‫report.
‫And now really a classification report is also a collection of different evaluation metrics rather than
‫a single one.
‫So that's where the report comes from.
‫It's kind of a report back of a number of different parameters evaluating our classification model.
‫Let's see.
‫One in action.
‫So from Skyline Dot Matrix Import Classification Report.
‫To do it.
‫We just posit some true labels.
‫Okay.
‫Y test and some predictions.
‫So once again, this evaluation metric is just comparing the true labels of our data versus the predictions
‫that our model has made.
‫So let's see what happens here.
‫All right.
‫There's a fair few things going on.
‫We got precision recall f F1 score support zero maybe that's for the class zero four, not heart disease
‫one.
‫Maybe that's for heart disease.
‫If we come down, we've got accuracy.
‫0.75 macro average.
‫Weighted average.
‫Hmm.
‫What's going on here?
‫Well, let's go to our keynote and have a look at our classification report Anatomy.
‫So over here, we see we've got the same thing.
‫Just import classification report.
‫We've done one on the test labels and the predictions, some slightly different numbers here.
‫What's in our notebook?
‫But that's okay.
‫So what's precision or precision indicates the proportion of positive identifications.
‫So a.k.a the model predicted class one, which were actually correct.
‫Now, a model which produces no false positives has a precision of one.
‫So that's where we can see our precision scores.
‫If the model produced no false positives, and we know that our model hasn't because we've looked at
‫the classification or the confusion matrix and we've seen it confused on some examples and it's predicted
‫them as false positives.
‫So our precision scores are a little bit lower than one.
‫Now the next one is recall.
‫So recall indicates the proportion of actual positives which were correctly classified.
‫Now, a model which produces no false negatives has a recall of 1.0.
‫And again, we know from our confusion matrix that this isn't the case.
‫So our recall is a little bit lower.
‫Now F1 score a combination of precision and recall.
‫A perfect model achieves an F1 score of 1.0.
‫Okay, so when we see our F1 score is not 1.0, but it's still relatively high, then if we go support.
‫The number of samples each metric was calculated on.
‫So we see here.
‫So this means we look at this 29.
‫So if we look at that, that's four zero and down here it's 61.
‫So that means that these metrics here were calculated on 61 different samples.
‫And these metrics along here will calculate it on 29 samples, which had the class labeled zero, and
‫that this one is 32 with the class label one.
‫Okay, has heart disease.
‫And so if you totaled these up 29 plus 32, you get 61.
‫Now, accuracy, we've seen that before.
‫This is just the accuracy of the model in decimal form.
‫Perfect accuracy is 1.0 aka our model is making predictions right 100% of the time.
‫If it's accuracy is 1.0, then we have a look at macro average.
‫Now what is that?
‫It's short for macro average, as you may have guessed.
‫So it's the average precision recall and F1 scores between classes.
‫Now the key here is macro average doesn't take class and balance into this should be effect into or
‫account.
‫So pretend this is a count.
‫So if you do have class imbalances, pay attention to this metric.
‫Okay, so in our case, what does it class imbalance?
‫Well, if you look at our example, we don't have class imbalances here.
‫Why is that?
‫Well, because we have relatively the same amount of samples with class zero and class one.
‫Meaning got about 5050 split here.
‫It's not exactly 5050, but it's not like we have 60 examples of Class one and one example of class
‫zero.
‫So we've got about 30 in each.
‫So if you did have class imbalances, you really want to check out your macro average now because our
‫example, the classes are relatively balanced.
‫That's why the macro average and the weighted average short four weighted average AVG, the weighted
‫average is a precision recall and F one score between classes weighted means each metric is calculated
‫with respect to how many samples are in each class.
‫So this metric will favor the majority class.
‫For example, we'll give a high value when one class outperforms another due to having more samples.
‫So this is why that our values here.
‫A quite similar because we have balanced classes.
‫Now you might be thinking is what kind of situation would it be that these values would be out of whack,
‫right?
‫Like where would this happen?
‫And you might also be thinking, we've learned a whole bunch of different metrics here.
‫We come back to our notebook.
‫We've gone over classification reports, we've done confusion matrix, we've done accuracy, we've done
‫area under ROC Curve.
‫So when should I use each of these?
‫So now we've kind of gone through them.
‫It might be tempting to think, Well, why don't I just ditch all these other ones that I really have
‫kind of trouble grasping and just use accuracy well.
‫Let's have an example.
‫Let's see when other metrics come into play.
‫And maybe you shouldn't just use accuracy because this is a trap that I got caught in right when I first
‫started building classification models, I was like, Yes, my model is getting 90 plus percent accuracy.
‫It must be a great model.
‫Let's see a scenario.
‫So for example, let's say there were 10,000 people and one of them had a disease, and you're asked
‫to build a model to predict who has it.
‫All right.
‫So let's actually do that.
‫Let's code this up.
‫So this is where precision and recall become valuable.
‫And in fact, all the metrics in our classification report become valuable here.
‫So disease.
‫True.
‫So there was 10,000 people and one of them had a disease.
‫So what we might do is create an umpire array of 10,000 people and then we might change one of these.
‫I can spell disease.
‫So disease zeroth index equals one.
‫So there's only one positive case.
‫Then if we come down here and say our model predicted or disease spreads equals NP Zero's 10,000.
‫Now this means model predicts every case as zero.
‫All right.
‫And then what we're going to do is create a data frame so we can visualize this classification report.
‫What that's what our model did.
‫Right?
‫This is what's happening here.
‫Disease.
‫True.
‫So there's only one positive case, one in 10,000.
‫And we build a model and our model predicts that every single case is zero.
‫So it misses the one prediction.
‫What would happen if we build a classification report so we go disease.
‫True.
‫We got over here.
‫Disease spreads.
‫I want to pass this little parameter so it fits nicely into a dataframe.
‫So output dict equals through.
‫All right.
‫So.
‫What's happening here.
‫We look at this, this is just another format of this classification report up here.
‫If we didn't do this.
‫It would error out.
‫So this is just to visualize what's going on.
‫True.
‫So this is a prime example, right, of where you want to use another metric other than accuracy is
‫when you have a very large class imbalance.
‫So in our case, we have a massive class imbalance because in our original data set, does these equals
‫true?
‫There's only one example, so only one example of where the label would be one and everything else is
‫zero.
‫We've built a model that just predicts zero for every case, right?
‫Because it just it's only one sample.
‫So it's really hard to learn that there's a pattern there for this one particular case.
‫And so what happens is if we were to measure just accuracy on our model that is predicted zero for everything,
‫it comes out with an accuracy of 0.99 or in other words, 99%.
‫And so ask yourself, although the model achieves 99.99% accuracy, is the model still useful?
‫Right.
‫That's why we look at these other metrics such as the macro average.
‫That's where we can see our models falling down here.
‫On the precision, it's actually getting zero precision for class one, which makes sense because it
‫didn't even predict class one.
‫And here it's getting basically 100% on class zero because it has only predicted class zero.
‫So this is a kind of use case scenario of where you want to make sure that you're using a wide spectrum
‫of evaluation metrics for your classification models and not just accuracy.
‫And now we've only just touched on some of the metrics that you can use for evaluating a classification
‫algorithm, but there are a few more.
‫So if you want to have a look at more, so we go here, model evaluation, socket, learn.
‫We'll look at the documentation.
‫If you come in here this is a little bit of extra curricular is to have a look at this section here.
‫So all I've done is gone into the socket loan documentation for model evaluation.
‫We come to classification, we can see a few here, the ones that we've covered.
‫And so the ones that we've covered are probably the bare minimum that you want to use, right?
‫So accuracy area under the rock curve or AUC confusion matrix and Classification Report.
‫So keep those in your tool bag and if you need more, be sure to refer to the documentation.
‫So that's going to wrap up our classification specific evaluation metrics.
‫Let's dive into the next section where we'll have a look at some more specific regression model evaluation
‫metrics.