‫All righty.
‫So we found a logistic regression model that performs pretty well, but we went and showed that to our
‫boss, and then she said we had to find a whole bunch of things like parameter tuning, feature importance,
‫confusion, matrix, cross validation.
‫And these things could sound made up to someone who's not a budding data scientist or machine learning
‫engineer like yourself, but being the budding machine learning engineer and data scientist that you
‫are, you know that data scientists make up words all the time.
‫And the good thing is that these aren't made up.
‫These are things that we can actually find.
‫So now we've got a baseline model.
‫And we know a model's first predictions aren't always what we should base our next steps off.
‫What should we do?
‫If we remind ourselves of the classification metrics and regression metrics, we're not worried about
‫regression, right?
‫Because we're working on a classification problem.
‫We've used the default accuracy.
‫Then there's precision, there's recall, and there's F1, which are all part of the things that the
‫boss asked us when we go to Confusion Matrix.
‫Okay.
‫We should probably make one of them since we're working on a classification problem.
‫Then we've got a classification report which has a bunch of information here like precision recall F
‫one score support, accuracy, macro average, far out.
‫And then we've got hyper parameter tuning.
‫We've covered before, but we've got a lot of steps.
‫So first we should put down what we're going to tackle.
‫Let's look at the following because we're a part of the experimental phase.
‫That's what we're going to be working towards.
‫We're going to be taking our models that we've got.
‫These are our baseline models here, and we're going to be experimenting with them further, seeing
‫if we can improve them and seeing does logistic regression perform best on accuracy, or could a tuned
‫or hyper parameter tuned version of a random forest beat it out?
‫Or maybe Knm could improve?
‫Who knows?
‫Let's write down a list.
‫Let's look at the following hypo parameter.
‫Tuning.
‫Feature important.
‫Confusion matrix.
‫Cross validation.
‫Precision.
‫Recall if one school classification report rosy curve area under the curve.
‫So as we saw in the socket learn section, these are some of the things that you should pay attention
‫to when you're working on a classification model.
‫These two here are actually part of almost any machine learning model that you'll be working on, but
‫these are specific to classification.
‫We've seen the difference between regression and classification matrix, but since we're focusing on
‫the classification problem, whether or not someone has heart disease based on their health parameters,
‫we're going to be focused on looking at these.
‫So.
‫Let's jump into it.
‫So the first thing is first, why don't we try hyper parameter tuning and cross-validation and remember
‫what's hyper parameter tuning?
‫Well, to cook your favorite dish, you know that to set the oven to 180 degrees and turn the grill
‫on.
‫But when your roommate cooks their favorite dish, they set the oven to 200 degrees and use the fan
‫force mode.
‫Same oven, different settings, different outcomes.
‫This is the same for machine learning algorithms.
‫You can use the same algorithms but change the settings a.k.a hyper parameters and get different results.
‫But just like turning up the oven too high.
‫If you do that for machine learning algorithms, you tweak the settings too much, you can change the
‫settings and it works well or so well that it over fits.
‫So we're really here.
‫What we're doing to do is look for the Goldilocks model, one, which does well on our dataset but also
‫performs well on unseen examples.
‫Should we churn a K neighbor's classifier?
‫Hmm.
‫Yeah, we might do that.
‫Even though we've said goodbye to the K and N classifier, we might just see how would you tune it?
‫And if you're wondering where I would start if I was trying to approach this, this is what I'd start
‫with.
‫How to tune a K neighbors classifier model.
‫We go here in depth parameter tuning for knm model selection, tuning and evaluation and came nearest
‫neighbours.
‫So if we were to read through these we would probably find some information that we're going to cover
‫here.
‫So let's start it out.
‫Let's make a little heading here.
‫We'll go here.
‫Hyper.
‫Parameter tuning.
‫Let's churn and.
‫Let's go a list of training scores and empty list.
‫So train scores and then a list of test scores.
‫Empty list.
‫Because what we want to do is compare different versions of the same model, so model with different
‫settings and compare their scores on the two different data sets.
‫Looking up what we looked at before, if we were to read into these again, you can do this with almost
‫any machine learning model or basically any machine learning model.
‫This is a part of the research.
‫This is a part of the experimentation.
‫Step of many machine learning problems is searching up things like this, then reading what you can
‫find and bringing it back to a scenario like what we're doing here.
‫But let's just pretend we've gone through that.
‫So we're going to create a list of different values
‫for and neighbors now Y and neighbors because as I said before, when we did our reading, we found
‫that if we go clean can.
‫Okay.
‫Neighbors classifier.
‫We see.
‫But neighbors is one of the parameters.
‫We could see this by reading through the documentation of psychic loan as well.
‫And so the number of neighbors used by default for cannabis queries.
‫In our research on how to tune a can model, we found that we can adjust the number of neighbors.
‫And if you want to find out more about this, you can check out the documentation.
‫You can read it up here.
‫So that's what we're going to do.
‫We're going to create a list of different values.
‫So the default here is five.
‫So maybe what we might do is try different values from 1 to 20 or something like that.
‫So let's see here, neighbors equals range 1 to 21, one, two, four.
‫And then we're going to set up.
‫K and N instance, so maybe K and n equals.
‫What is it?
‫K You know this k neighbors classifier beautiful.
‫And then we might go loop through.
‫We need to loop through different and neighbors.
‫Which is our ace that we made here for I in neighbors thinking out loud here we'll go k and dot set
‫params.
‫Remember you can adjust the parameters of a machine learning model in cyclone using dot set params equals
‫AI because AI is going to loop through this range here.
‫Wonderful.
‫And then we're going to fit the algorithm.
‫What we're trying to do here is improve on this baseline score.
‫So this is with the default of five.
‫And so we're going to try 20 different versions, 1 to 20 to see if it's any better.
‫That didn't do anything.
‫I accidentally hit shift enter and trigger happy.
‫You know, my fingers are just they just default to shift and enter whenever they want.
‫Now we want to update training scores list.
‫So we're going to go training.
‫We want train scores up here.
‫Train scores append.
‫Now we want.
‫How can an instance score on X train?
‫Y train.
‫Wonderful.
‫And then we want to update the test scores list.
‫What do that test scores append K and score.
‫And you can probably guess that this is going to be on the test set.
‫Beautiful bonus points if you did.
‫Now, what this is going to do is just going to loop through this range of 1 to 20 and then it's going
‫to create 20 different K and N models and append their scores to LIS.
‫So let's check out those lists.
‫Try and scores shift into wonderful and then go test scores.
‫Oop.
‫Typo.
‫Okay.
‫Now we've got 20 different schools or so, but these are probably best visualized.
‫Let's see how we would do that.
‫Let's go plt dot plot.
‫We're neighbors.
‫And we'll go train scores and then we'll go label equals train score.
‫Of course, yes.
‫Plus, he will add another plot.
‫Neighbors.
‫Test scores.
‫Label equals test score.
‫Wonderful.
‫Then we want to go plt.
‫Actually, let's just see what this looks like.
‫X label.
‫We'll go number of neighbors because that's what's on the x.
‫I'm spelling neighbors the Australian way here with the u there.
‫Or maybe it's not Australian.
‫Maybe it's just a different version.
‫But neighbors in a psychic line is spelt with no u.
‫So this is the x and then plt dot y label is going to be the scores.
‫So this is x y.
‫So this is going to be model score.
‫And then we want plt dot legend and we'll see what comes up.
‫Actually, we might.
‫A little tidbit here, Max.
‫If so, maximum can score on the test data.
‫We're going to go, Max.
‫Test scores, it's going to be accuracy.
‫So we want times 100.2 F close parentheses and then close string.
‫That should work.
‫Wonderful.
‫Okay.
‫So we can see here that we're really paying attention to this test score here so we get the highest
‫value here.
‫He's around about 11 for four K nearest neighbors.
‫So if we look at the graph, actually, that's.
‫Let's adjust these x ten so we can see where it actually is.
‫X Text MP range 1 to 21 space of one.
‫So what this is going to do is just bring out a list of range of 1 to 21 of space one.
‫Which is the exact same as what we're using here.
‫So neighbors vehicles range 1 to 21.
‫So let's do that.
‫And yes, our assumption before before we adjusted the X ticks, which is these just little labels down
‫here, was that a K, an N or a neighbors value of 11 yields the best score on our test data set.
‫So if we come up here, the default we remember is five.
‫So we've just done a little bit of hyper parameter tuning and we were able to improve our K and classifiers
‫results on the test data set by changing it from the default five and NW neighbors parameter.
‫So we're doing hyper parameter tuning here.
‫Change that to 11 and we get 75.41 versus our initial result, which is back up here of 68%.
‫Hmm.
‫Well, even then, even after improving it to 75 or whatever it was, it's still far below our logistic
‫regression and random forest.
‫So I think this has put the nail in the coffin for our K and N model.
‫Is that because even with a hyper parameter tuning, it still hasn't reached the scores that we got
‫with logistic regression or random forest?
‫And so because of this, we're going to discard K and for now this is part of our experiment, a big
‫part of experimenting in machine learning.
‫Step six is going through different machine learning models and figuring out which work and which don't.
‫So what we're trying to do here is as quickly as possible, work through different little experiments
‫like what we've just done, tuning N and seeing which model performs best on our data.
‫So what we might try and do is we've churned Kanon by hand like we wrote out this little for loop here.
‫That was a bit tedious, right?
‫So you had to do that for every single machine learning model.
‫You might run into some problems.
‫It was okay here because we had one parameter tune, but if you had more parameters, writing these
‫for loops is going to be very inefficient.
‫So what we might do next, you see how we can churn, logistic regression and random forest classifier
‫using randomized search CV, which stands for Cross-Validation.
‫So instead of us having to manually try tuning different type of parameters by hand randomized search
‫CV, which we've seen in the second line section, it's going to try a number of different combinations
‫of hyper parameters for us and evaluate which ones are the best and then save them for us.
‫So that's what we're going to have a look at that next video.