‫So let's start diving into different ways we can improve our models using hyper parameter tuning.
‫The first one here is by hand.
‫So let's make a little heading 5.1 tuning hyper parameters by hand.
‫And so far we've talked about dealing with training and test data sets.
‫A model gets trained on the training set, it finds patterns and then it gets evaluated on the test
‫set.
‫So it uses those patterns, but hyper parameter tuning introduces a third set, a validation set.
‫So if we have a look here, this is what's going to look like when we tune hyper parameters by hand.
‫So if we were starting with 100 patient records in the case of our heart disease problem, now again,
‫this this may change depending on what problem you're working with.
‫And I've just used the number of hundred because it's an easy number to picture.
‫So this is our starting data, and what we might do is split it.
‫This is what I mean by three different sets.
‫So we split, we might use 70 to 80%.
‫So 70 patient records as a training split, we might use 10 to 15% as a validation split and we might
‫use 10 to 15% on a test split.
‫Now, of course, you can adjust these numbers.
‫These are just some common numbers you'll see used.
‫So naturally the model gets trained on.
‫These is what we've seen before.
‫And usually without the validation set, our model will get evaluated on these.
‫But now, because we have a validation set, this is where we're going to tune our model settings aka
‫the hyper parameters.
‫Get tuned on the validation split and then finally as normal a model gets evaluated on the test split.
‫And now the analogy here is, if you remember right back at the start, the most important concept in
‫machine learning is the three sets.
‫So this was right back in the start.
‫Machine Learning 101.
‫The training set is analogous to say you were at a university course and you were learning the course
‫materials.
‫The validation set is where you would test your knowledge a little bit and see what you need to adjust.
‫So you would get the practice exam from your professor and you try it out and you go, Oh wow, I'm
‫doing terribly at questions one, two and four.
‫Maybe I need to adjust the way I approach this.
‫And then you do the practice exam again.
‫And once you'd improved your skills on the practice exam, you'd feel a little bit confident and you'd
‫finally really evaluate yourself on the final exam, which is the test set.
‫So how would we do this with code?
‫Let's come back to our notebook.
‫Tuning hyper parameters with hand.
‫Lets make three sets.
‫Training, validation and test.
‫So what we'll do is we'll just remind ourselves of our get params or remind ourselves of our random
‫forest classifiers.
‫Baseline parameters.
‫So these are all the baseline parameters.
‫Okay.
‫The settings on our model which we can adjust.
‫And now, in fact, which ones are we going to adjust?
‫Well, after reading the Random Forest documentation and again, you can do this for any machine learning
‫estimate or or model with psychic loan, we start to get an idea of how we could adjust each setting.
‫And even with some of the models, there'll be some nodes on different hyper parameters.
‫Psychic, learn, suggest to change.
‫That is kind of like a one.
‫Do you want to change first to try.
‫And so after reading through that, after going through that range, so in our case, we're going to
‫try and adjust the following.
‫So we want max depth.
‫Actually, we'll put these in back ticks because we know that they're so we know that they're code max
‫step.
‫Max features.
‫Mean samples.
‫If none of these make sense, remember, the definition to all of these is in the documentation for
‫any model.
‫So if we come up here means samples leave the minimum number of samples required to be at a leaf node
‫and you can read more in depth in what a leaf node is if you check out some resources on a random forest
‫in depth.
‫But for now we're just focusing on how we can adjust.
‫Hyper parameters of a machine learning model means samples leaf.
‫And mean samples split and an estimate is wonderful.
‫So these are the high parameters we're going to adjust.
‫And now we'll use our same code as before, except this time we need to create a training, validation
‫and test spline.
‫We'll just use these these splits here.
‫So the training split will create with 70% of the data.
‫The validation and test sets will each contain 15 and we'll get some baseline results.
‫So we've kind of already got them, but we'll do it again, get some baseline results, and then we'll
‫see how we can tune the model type of parameters by hand.
‫And since we're going to be evaluating a few models, I think it's important that we create an evaluation
‫function.
‫So this is what you might want to do.
‫Whatever model you're you're working on is create functions.
‫If you know you want to do something more than once.
‫So it saves out writing lots of code.
‫But again, that's probably a bit different to what we've seen in the past.
‫Right.
‫We know we like to write everything out by hand.
‫We'll just leave a little docstring here.
‫It performs evaluation comparison on y true labels versus y pred labels.
‫And now, as we saw in the evaluation section, the main thing evaluating machine learning model does
‫is it compares its predictions versus the true labels.
‫So that's what this function is going to do.
‫So because we're working with a classifier, we want accuracy, accuracy, score y true y spreads.
‫We also want precision.
‫And now we might actually note here on our classification model, because if we use this evaluate spreads
‫on a regression model, we're putting in classification metrics, we're going to get errors because
‫a regression model predicts different things to what a classification model predicts precision.
‫And then we want recall, of course, equals recall score.
‫And now again, you could adjust this, evaluate Fred's function for what you need.
‫But I'm just going to include some of the most common metrics that we've covered for classification
‫models, some of the things you want to pay attention to for basically all of your classification models,
‫Y spreads, and then we'll create a dictionary so it can return the predictions, so we can compare
‫them with other predictions.
‫Later round accuracy.
‫We'll go to two decimal places.
‫Yep.
‫Precision.
‫Round precision.
‫To.
‫Now, always remember, if this seems like a lot, it's because, first of all, it kind of is a lot
‫to take in one hit.
‫And the second thing is that I've had a fair bit of practice with this, so I inherently know what to
‫use.
‫Again, I'm always learning, but I've just had a bit of practice building these specific kinds of systems,
‫and these patterns come up time and time again.
‫We go print.
‫We want just to give us a little bit of a printout of what's going on.
‫Ach, we'll go accuracy.
‫Time's 100 and then we only want two decimal places.
‫That will be enough.
‫Then we'll go to the same thing if precision.
‫This can be precision.
‫We won't need to times that by 100.
‫That can just stay at two decimal places.
‫Well, we need a string ender.
‫Wonderful print.
‫Then we're going to go recall.
‫This is a lot to type out to begin with, but because we'll be able to reuse it, it's going to save
‫us down the track.
‫And then finally we want print.
‫If one school.
‫We're making it.
‫We're finally making it.
‫Excellent effort.
‫There we go.
‫And finally, we want to return our metric dipped.
‫Now, what is happening here?
‫Well, essentially, this function just takes some true labels and some prediction labels from our classification
‫models.
‫We know this from the string here.
‫It's going to compute different evaluation functions here, and then it's going to return a metric dictionary
‫so we can save that for later.
‫And it'll also print out some metrics.
‫So we get a print out as soon as we run this function.
‫So it's just an evaluation function for our classification models.
‫Wonderful.
‫And I'm going to save the notebook here, as always.
‫Now, how exactly would we create these train validation and test splits?
‫Well, our train test split function that we've seen before, remember this one?
‫Train to split.
‫So this one only returns, train splits, arrays of matrices into random train and test subsets.
‫So this one only splits into train and test sets.
‫So what we'll need to do is we need to manually split our data into train validation and test sets.
‫And so to do this.
‫We can.
‫How can we do this?
‫Because our data is in a data frame.
‫Heart disease.
‫We should just be able to do it using some good old math and indexing.
‫Let's see how we would.
‫So from Skyline, we'll import our model first, even though we've already instantiated a random forest
‫classifier.
‫Let's.
‫Go EMP random seed and then we'll shuffle the data.
‫Why are we shuffling the data here?
‫If we're splitting into train validation and test split.
‫We want to mix it up.
‫Make sure it's not just the same.
‫The records are coming in order that they come in.
‫If we're using slicing to create our train validation and test line, we want to make sure all of these
‫records are jumbled up.
‫So that's what we'll do.
‫We can shuffle it using PANDAS sample function.
‫We'll do frac equals one for 100% of the data.
‫This is just going to take the heart disease, sample it.
‫This is randomly and then reassign the heart disease data frame to its normal variable.
‫So it'll just take this sample a bunch of rows at random and then reassign it to this variable, a.k.a.
‫shuffling the data.
‫Now, now the data has been shuffled, will split into x and Y.
‫Actually, we might save that to heart disease shuffled.
‫That's a better idea.
‫X equals.
‫This time we need heart disease shuffled and then we're going to drop the Target column here.
‫This video is dragging a little bit on, but this is an important concept, right?
‫We're going to see here.
‫It'll all be worth it because we'll be able to improve our models using hypothermia to tuning, split
‫the data validation and test sets.
‫Wonderful.
‫So the train split.
‫How are we going to do this?
‫Well, we need to create a number that we can use for slicing.
‫So we want 70% of the length of our data.
‫So you see what's happening here.
‫Try and split.
‫We you need a number.
‫We need 70%.
‫So 0.7 times the length of heart disease shuffled and this is going to be 70% of data.
‫And then we need a valid split.
‫Which is going to be.
‫Do you remember?
‫Do you remember what the split is?
‫15%.
‫That's what we've agreed on.
‫Wonderful.
‫So we can do that in the same way.
‫That will be train split because we want to index.
‫We want the next 15%.
‫So train split plus 0.515.
‫The length of our heart disease shuffled.
‫DataFrame.
‫This is 15% of data.
‫Wonderful.
‫And so X train, y train is going to be the X data.
‫Up to the train split.
‫And the why data up to the train split.
‫Wonderful and then ex valid y valid sorry is going to be the x data from the train split to the valid
‫split.
‫And the same thing for y is going to be the y data from the train split.
‫To the valid split.
‫And then we're going to have X test is going to be and y test is going to be the X data from the valid
‫split onwards.
‫So the rest of the data.
‫And then why from the valid split onwards.
‫Whew, that was lost so far.
‫Let's just see what's going on.
‫Right, let's let's go, Len, print it out.
‫Ex train and then we want Len ex valid.
‫And then we want len ex test.
‫Okay.
‫So the training set has 70% of the data, aka 212 rows.
‫The validation set has 46 rows, 15% of the data, and the test set has 46 rows.
‫So that's 15% of the data.
‫There's different amounts here, even though they're both 15% because we've used round we have an odd
‫number of samples in the heart disease shuffle data set.
‫So there's going to be a spillover of one somewhere.
‫That is perfectly fine.
‫Okay, now we have our splits.
‫We can do KLF, we'll instantiate a random forest classifier.
‫And now, because we've passed nothing here, this is going to instantiate it with the baseline parameters.
‫So if we do this CLF dot get params.
‫Oh, that's not an attribute.
‫That's a function.
‫So without passing anything to here our random forest classifier and instantiate the random farm classifier
‫with the baseline parameters aka this.
‫And so that's what we want because we want to make some baseline predictions.
‫So we go see left fit.
‫We're going to fit it on the training data as usual and then we're going to make predictions.
‫So y spreads equals self predict.
‫No, we're going to predict on the validation data because we come back here, we want to tune our model
‫on the validation split.
‫So that's what we're basing our metric on is will first create a baseline metric, which is by running
‫our evaluation function on the validation split, then we'll adjust the hyper parameters and try our
‫model again on the validation split and see how they compare.
‫So let's go here.
‫So make predictions.
‫We'll call this baseline predictions and then we'll go evaluate the classifier on validation set.
‫So we'll go baseline metrics.
‫And now this is where our evaluation function will come in handy metrics equals evaluate spreads y valid
‫so see how we passed it.
‫The why the validation set and why Preds.
‫And then we want to go baseline metrics.
‫Well, what have we got?
‫Local variable accuracy.
‫Reference before.
‫Oh.
‫This is what we've missed out on.
‫This needs to be accuracy score from this again.
‫It's done terrible now.
‫What is going on here?
‫Accuracy point 22.
‫Have we done this correctly?
‫Oh, there we go.
‫That's why we've used our original.
‫See, that was like that.
‫We need to use heart disease shuffled because otherwise our model was predicting the original labels
‫rather than our shuffled labels.
‫So that's that's all we missed out on there.
‫There we go.
‫Woo!
‫So we're getting this warning here that the value of an estimate is will change from ten inversion,
‫0.20 to 100.22.
‫So this is our baseline metrics.
‫Now, what we've done is we've created a train validation and test split.
‫We've instantiated model just if we had done before, we fit it on the training data here.
‫And then we've evaluated the baseline parameters, baseline hyper parameters on the validation set and
‫got these metrics.
‫So if we were to try and improve our results, if we were to try and adjust a model's hyper parameters.
‫These ones here on our random forest by hand, because that's what this section is.
‫Tuning higher parameters by hand.
‫How would we do so?
‫Let's take this warning as an example of changing a hyper parameter.
‫Let's change.
‫And again, if you're using psychic loan 0.2, two or above, you won't get this warning.
‫But since we are we're getting this warning, we're going to try and adjust.
‫Our hyper parameters will change an estimate us which is this one here.
‫We'll change it from the baseline of 10 to 100 and see if we get a different score on the validation
‫set.
‫Create a random seed.
‫Random Seed 42.
‫What we're going to do is create a second classifier with different hyper parameters because remember.
‫Instantiating our random forest classifier up here.
‫Passing it nothing.
‫Instantiate the classifier with the baseline hyper parameters it comes with right out of the box.
‫So like using your oven's predefined settings when it comes out of the box to cook your favorite dish.
‫These scores aren't really helping us out here.
‫We want a better model.
‫We'll create CLF two and we're going to adjust the hyper parameters like you would adjust your oven
‫to try and improve that favorite delicious roast chicken dish that you're making.
‫Maybe you're preparing for a big function that you're all your friends are coming over and you've promised
‫them a great dish, so you're trying to perfect it.
‫That's what we're doing with our machine learning model.
‫So again, same thing, same data fitted on the training data, but this time we've got CLF two, which
‫is using an estimated as 100 rather than ten.
‫And now again, we could try different settings for all of these features here, the ones that we're
‫going to try and adjust.
‫But we might start with just one, see an example of it, and then we'll go make predictions.
‫So why Preds, too?
‫Because we're using CLF to CLF to predict.
‫X valid.
‫We're doing the same thing as up here.
‫The baseline predictions are on the validation set and now make predictions with different hyper parameters.
‫On the same data.
‫So different models, same data.
‫And now we're going to evaluate the second classifier.
‫CEO left two metrics equals we're using our valuation function again already the one we created before
‫and we're saving a fair few lines of code by just calling it like that.
‫All right, now let's check it out.
‫Ah.
‫Okay.
‫So now if we compare using the baseline hyper parameters that our model came with, what can we see
‫that's different here?
‫All we've changed is an estimate as equals 100.
‫So we've adjusted one dial on our model.
‫So like adjusting one dial on your oven, we can see a slight boost in accuracy on the same data.
‫So CLF two has a slightly higher accuracy.
‫It's got a higher precision.
‫But a lower recall and at the same f one score.
‫Hmm.
‫Okay.
‫That's giving us a little inkling that maybe if we kept going with different high parameters, we kept
‫trying to adjust them by hand.
‫We would eventually just hopefully keep improving these metrics.
‫So what would you think we would do next?
‫If we go up here, maybe we try to to change the max depth.
‫So what's the default max depth?
‫We find it in here.
‫Max depth is none.
‫So maybe we'd look at the documentation and go to the different variables that max length can take so
‫it can take integers or none.
‫Default is none.
‫The maximum depth of the tree and doing our research, reading the documentation, we see some different
‫values for max depth.
‫So then we'd go back to our model here and maybe we create clef three equals random forest classifier.
‫This time we'll keep an estimate as as the same equals 100 and then we'll change max depth from none
‫to equal ten.
‫Now.
‫I haven't made this number up.
‫I've done some research, read the documentation and figured out different ideas for Max Depth, and
‫then we might do the same thing as what we've done here.
‫Evaluate our third classifier, as you might have guessed, or as you might have thought, if we're
‫going through and adjusting all of these by hand, adjusting all the knobs by hand, that's going to
‫take a fair bit of work.
‫Right?
‫So just like perfecting your dish in real life, like if you're making a favorite chicken dish, everyone's
‫coming over for dinner, you're trying to adjust the settings on your oven.
‫You may have to go through a bit of trial and error, right?
‫You might cook it once and then it's not that good.
‫And then by the 10th time, you're starting to get really good.
‫But that could take a lot longer with what we're trying to do here, writing code, if we have all these
‫different settings up here, trying to change these, trying to find the best settings it could take
‫far longer than what we have.
‫And the rule is in code.
‫Don't repeat yourself.
‫So you might might have guessed psychic learn has a way, a method inbuilt that can do this for us.
‫Try a bunch of different settings for us and that's randomized search save but we're going to have a
‫look at that in the next video.
‫This one's already getting far too long, but the main takeaway here is that when we're tuning a model's
‫hyper parameters, you can think of the training set as like the course materials where you're you're
‫learning the foundation data, that's the training set.
‫And then on the practice exam, you're refining what you know.
‫So you've already learned all the baseline patterns like our model.
‫And then with the validation set, we're adjusting the settings.
‫So we're refining what the model knows, how it learns different things before we test ourselves on
‫the final exam.
‫So in a.k.a before we evaluate our model on the test set.
‫Now, don't worry.
‫If you're feeling a little bit overwhelmed, go back through the code that we've written.
‫Check it out.
‫You could try out adjusting a few different parameters yourself, but if not, not to worry.
‫We're going to see in the next video how we can use a module in psychic.
‫Learn to try different hyper parameters for us.