‫Who?
‫I'm rubbing my hands, if you can hear that.
‫You know why?
‫Because I'm pumped.
‫I'm ready to.
‫To evaluate our machine learning models or see how they go.
‫See how each of these models using the function we've just created in the last video.
‫See how logistic regression goes.
‫K Nearest neighbors classifier and random forest classifier goes at finding patterns in our training
‫data and then how those patterns get evaluated in our test data.
‫So without any further ado, let's see how each of these models performs so we can call our function
‫fit and score and hopefully it works.
‫And we got here.
‫We see that it takes models or actually if we press shift tab, this is where our docstring comes in
‫handy because it tells us what happens, fits and evaluates given machine learning models.
‫This is beautiful.
‫This is just like what we've seen with other functions.
‫But except we've created this one ourself, that's a helpfulness of a docstring.
‫Right?
‫So we're going to go here.
‫Models equals models, which is our dictionary of machine learning models.
‫We could just go x train x test, but for completeness, we're going to go x train equals x train and
‫then we're going to go x test equals x test and then we're going to go y train equals y train.
‫And then finally y test equals y test.
‫And if we've written a function correctly, what do you think it's going to return?
‫We'll give you a few seconds.
‫Okay, that's enough because we want to see it.
‫If in doubt, run the code.
‫So let's see what the R function returns.
‫Okay.
‫So we get some warnings here.
‫What's it say?
‫Stop total number of iterations reached.
‫Hmm.
‫That would require us to dive in, increase the number of iterations.
‫So this is where it's helpful, increase the number of iterations, max beta or scale it down.
‫So this is saying that potentially our logistic regression model could be improved.
‫And to figure out how you could do that, it would require going to the documentation for alternative
‫solver options.
‫But what we're going to do is just work with what we've got so far.
‫So if we have a look at this, this is the score.
‫How each of our models without tuning, well, that's a spoiler for what's coming out.
‫This is how each of our models as the baseline class has performed on finding patterns in our test data.
‫So it's getting the score here.
‫This is what we're getting back.
‫So if we look at this, which ones are highest, look at that logistic regression coming in as the dark
‫horse, not even on the machine learning map.
‫Right.
‫And it's getting the highest score.
‫Hmm.
‫Well, this might take a little bit of investigation, because remember where we up to?
‫We're up to experiments and that's what we're doing here.
‫An experiment is comparing different models, trying different models, and comparing them to each other.
‫Now, this is a comparison in a dictionary, but if we wanted to show this to someone, we might compare
‫it visually.
‫So let's do that nice and quickly.
‫We can make a model.
‫Actually, we might make a little heading here.
‫So we go here.
‫Model comparison.
‫Beautiful sir model compare equals might turn our dictionary into a dataframe because that's nice and
‫simple.
‫So model scores.
‫This is just taking this dictionary here.
‫Model scores.
‫And we're going to set the index to be.
‫What does it have to be?
‫It has to be a list called accuracy, because that's what this score function here, because our models
‫are all classifiers, their default score metric is accuracy.
‫We saw that in the psychic loan section.
‫So if we go here model compare, we need to transpose it plot dot, bar, boom.
‫Now I'll just show you why we need to transpose it because if we did that, it would look weird like
‫that and we actually want it to look.
‫Good luck that.
‫So this is this is quickly showing how accurate each of our different models are.
‫And you can see logistic regression just tips out our random forest and can well because that's nowhere
‫near the accuracy of our logistic regression model.
‫We're going to say goodbye to k n.
‫So once we've got this graph, this might be something that we can take to the boss or one of our colleagues
‫and say, Hey, look at this.
‫We've built a machine learning model, a beautiful logistic regression model, and it's performed the
‫best.
‫So we're going to use the logistic regression model in practice.
‫And so you might go, I found it, and your boss is like, Nice one, what did you find?
‫And then you're like, Well, the best algorithm for predicting heart disease is logistic regression.
‫And then she might say something like, excellent.
‫I'm surprised the high parameter in a tuning isn't finished by now.
‫And then you might wonder what is hyper parameter tuning?
‫Yeah, me too.
‫It went pretty quick.
‫So you sort of covering yourself here.
‫And then she might say, Well, I'm very proud.
‫How about you put together a classification report to show the team and be sure to include a confusion
‫matrix and the validated precision recall on F1 scores.
‫I'd also be curious to see what features are most important.
‫Oh, and don't forget to include an ROC curve.
‫And then you might be thinking or asking yourself.
‫Okay, that's a that's a lot of complex words there.
‫Great.
‫But then you actually say, of course, I'll have it to you by tomorrow.
‫Don't worry, we're going to take care of all these things.
‫We're going to take care of all these things in the next video, because we're still at a point where
‫even though we found a great machine learning model and we've probably shown it to the boss or one of
‫our colleagues, and they've gone, Wow, let's just do a regression.
‫Performing at 88% accuracy.
‫We're still not near our valuation metric, which is if we come back up the top.
‫Which, as we said, we want at least 95% accuracy to continue with this experiment.
‫So that's what we're going to be doing, as well as fulfilling all those requests at the boss asked
‫us just to make sure our model is a little bit more robust than just getting the default score metric.
‫So without any further ado, take a little break, have a review of what we've done so far.
‫And we're going to review some of the things that we need to make sure we do to make sure our classification
‫models are evaluated correctly and are improved as well as possible.