‫Welcome back.
‫And the last video we saw, how to tune a model's hyper parameters using randomized search.
‫CV And what that did was we defined a space or a dictionary of different hyper parameter values.
‫We'd like to try and then randomize search.
‫CV Combine these parameters at random and built a classifier with different settings, different hyper
‫parameters, and then evaluated them all using cross validation.
‫Or we might change this back to ten.
‫We don't want 50 models running off.
‫And then after trying ten different combinations, it found the best parameters to be this set of values
‫here.
‫Then after evaluating them, we saw it didn't really do as well as our previous model up here, where
‫we churn just by hand.
‫And so finally, a workflow you might end up with when you're tuning high parameters is starting off
‫by hand and then using randomized search CV across a space of high parameters.
‫And then once you've found some pretty good hyper parameters, you'll probably finish up with grid search.
‫CV Let's see that in action here.
‫So 5.3 hyper parameter tuning with grid search CV.
‫Beautiful.
‫And let's check out our grid.
‫So our grid at the moment is you might notice the grid search curve has the word grid in it.
‫And our grid of hyper parameters at the moment are these values here.
‫And the key difference here between randomized search and grid search is that randomize search has a
‫parameter called NITTA, which is what we saw up here.
‫Which we can set to limit the number of models to try.
‫So in our case, we used ten.
‫So if we look back up here, the top of this code fitting five folds for each of ten candidates totaling
‫50/5.
‫The key difference between randomized search CV and grid search CV is that grid search CV is kind of
‫like a brute force search.
‫It means it will go through every single combination that is available here.
‫So if we have a look, we've got six values there.
‫So it'll be six times, five times two times, three times three.
‫So there's six parameters here.
‫There's five here, two here, three, three in total, 540 different parameters that it would try or
‫combinations of parameters.
‫And that's a lot that's an then cross validated.
‫So time's up by five so 2700 models that's a lot, right?
‫That's going to take a lot of compute power training.
‫One model might take long enough time, but training 2700 of them, especially if you're working with
‫a large data set, probably not going to be suitable for your laptop.
‫You may need a big computer or we can reduce the search space.
‫So let's see that in action.
‫So what I mean by that is we'll create Grid too, using a little bit more of a confined hyper parameter
‫space.
‫So in essence just reducing the number of hyper parameters grid search has to go through.
‫So let's see this in action.
‫What we might do is just copy this.
‫And now how would you create Grid two in practice?
‫We could just copy and paste it, which we've done.
‫I want to line these up so it all looks polyphonic.
‫There we go.
‫Now we could just copy and paste it and then just delete these at random.
‫Delete.
‫We might not do that.
‫So we've gone through Randomized Search XXV and it's gone through.
‫Let's say we up this to 50 different models like we had before, but in our case, we've only actually
‫done ten.
‫We might take these parameters.
‫The best params from our randomised search CV.
‫And use them to influence where we'd like grid search to run.
‫So let's do that.
‫Our best parameters is 262 square root.
‫So what we might do?
‫Remove ten.
‫Keep 500 there and remove these two.
‫Yep.
‫Max depth is none, so we might just keep that as none.
‫So it's only got one option there.
‫And then for square root we might ah actually keep that as, as auto or square root.
‫And then what else do we have.
‫Mean sample split is six.
‫We might get rid of four and two and then what we might do min samples leaf.
‫So we might keep that as one and two.
‫So what have we done here?
‫Well, what we've done is we've reduced our search space of hyper parameters.
‫So now.
‫Remember we had 540 different combinations there.
‫What do we have?
‫So we have three times.
‫One times two times one times two.
‫So 12 and then it's going to be cross validated five times so we can adjust this using the KV parameter.
‫So 60 versus 2700.
‫So that's a lot less parameters there.
‫So how would we do this in practice?
‫How would we use grid search CV to find the best hyper parameters in this space?
‫Well, as always, let's see it in action from Skyline DOT Model Selection, Import Grid, Search CV
‫and we want to import as well train test split.
‫We probably already have that.
‫We can put that there.
‫Random seed.
‫Wonderful.
‫Split.
‫Edit What we might do actually is just bring this code up here because it's the exact same code that
‫we want to use before.
‫So we'll just take this.
‫So we're going to change one thing.
‫Come down.
‫Wonderful.
‫So we're using the same dataset as our randomized search CV, making them the same split.
‫We're instantiating a random virus classifier, but this time we're going to set up.
‫Grid Search CV.
‫Wonderful grid search CV.
‫And then we're going to fit it.
‫The grid search.
‫Save it here and now.
‫There's a different parameter here.
‫So this one is going to be param grid, I believe.
‫It's not going to have an ETA because grid search is like brute force.
‫It tries every single combination and we want it to try our grid to not grid one because it's got a
‫few less options there in grid two as we saw before.
‫So we'll come down here keep cross-validation is five fold the BOEs equals that okay now let's step
‫through what's happening.
‫So we're importing grid search CV.
‫We've created grid two, which is a refined search base of different hyper parameters.
‫So different settings, so much like if you were working on your favorite meal, adjusting your oven,
‫if you set the temperature to 400 degrees, that's way too high.
‫So if it starts off at 180 degrees and it does, okay, you might go, why would I go right to 500 or
‫400 degrees when I can just make a little jump to 400?
‫So that's why we've created Grid two.
‫We've based it off the best parameters or the best hyper parameters that our randomize searched CV or
‫the randomized search CV has found.
‫So let's do it.
‫Let's build a random forest classifier using grid search CV.
‫So here's how it's going to go again.
‫Same thing.
‫And fitting five folds for each of 12 candidates totaling 60 fits.
‫Now what grid search Seavey is going to do, just like randomize search.
‫CV It's going to go through all of these different hyper parameters and pass them to our random forest
‫classifier, trying each different combination.
‫And then eventually when it's finished, we'll be able to go, Jeez, CLF best params.
‫Wonderful.
‫So it might take a little while with grid search.
‫CV This is something to keep in mind.
‫The more hyper parameters you have in here, the longer it will take grid search.
‫CV To run because it has to try every single combination of hyper parameters in here.
‫So now, as we did before, we can evaluate our grid search classifier by making predictions with it
‫and then using our evaluation function to figure out how it did.
‫So let's do that.
‫Clf I actually will go.
‫Geez, why?
‫Preds equals gs.
‫CLF predict will predict on the test data as well.
‫X test and then we'll go evaluate the predictions.
‫GPS metrics equals evaluate spreads.
‫Our function, one that we defined in a couple of videos ago and then we'll go, geez, why Preds?
‫Wonderful.
‫And now again, we see a slight decline in accuracy.
‫Hmm.
‫And that just goes to show how much of a process tuning hyper parameters can be.
‫Right.
‫It's all about that trial and error.
‫What you could do to fix this is go back up.
‫Your workflow would probably be try a few settings by hand as we did right back up here.
‫Tuning high parameters by hand.
‫This is the workflow you'd probably do.
‫Try a just a few of these by hand.
‫See if you can figure something out with a validation set like we've done and then try some high parameter
‫tuning using randomized search.
‫CVE.
‫You might, in this case, up this number to 20 or 50 or so or whatever arbitrary number that you find.
‫Again, this is all experimentation to try and figure out what type of parameters are better.
‫And you might even change the grid that you're searching over based on the hyper parameters you find
‫in the cyclone model documentation for the model that you're using.
‫And then once you've found some good hyper parameters using randomized search KV, you might take that
‫those hyper parameters, these ones here best params and then create another grid to grid search over
‫like we've done here.
‫In our case, it hasn't improved our model's metrics, but this is where we'd probably start to experiment
‫a little bit more.
‫Try a few different other metrics, try a few different other hybrid parameters.
‫But for completeness, what you'd probably do then, after you've tried a fair few different hop parameters,
‫you've created a baseline model with the default settings.
‫You've tried to adjust them by hand, you tried to adjust them by randomize search TV, and you've tried
‫to adjust them by grid search.
‫You're going to want to compare your models.
‫So let's compare.
‫Our different models metrics.
‫Wonderful.
‫And to do so, we're going to create a dataframe.
‫So.
‫PD This is why we've been returning dictionaries from all of our, all of our models and our evaluate
‫pred function.
‫So baseline is going to be baseline metrics and then CLF two is going to be self to metrics.
‫I've been here when we don't need them.
‫Random search is going to be a race matrix, and then grid search is going to be GS matrix.
‫Wonderful.
‫And then to see it in action, we're going to go compare metrics, which is just our data frame that
‫we've instantiated here.
‫Well, we might need to move.
‫It's up there.
‫It's going to be an error there in there that's actually squiggly bracket that we didn't need.
‫Plot dot bar.
‫Fig size equals ten act.
‫Let's see it.
‫Wonderful.
‫So this is going to compare all of our different classification models metrics in the one space.
‫So we can see our baseline model is the blue.
‫So on this one, it didn't get the best here.
‫It was more of a tie between CLF two and random search.
‫On precision, CLF two is out in front.
‫On recall, baseline is out in front and on F1 it's a tie between baseline and clef two.
‫And again, you could adjust this graph to make it more visible.
‫This is just to demonstrate different metrics.
‫So this is the kind of communication you do for not only yourself, but for your teammates.
‫Or if you're reporting to someone in a project, is to show them the different models that you've tried
‫and then how they perform differently on different classification metrics.
‫If you're working on a classification problem, of course you can do the same sort of process with a
‫regression problem.
‫And now, depending on what you need for the project, will depend on which model.
‫So say you want a model that optimizes recall.
‫In this case you might use the the baseline model or if you wanted to model that, optimize precision,
‫you might choose the self to model.
‫And if none of these metrics really worked for you, you might continue to try to find some better ones
‫through random search or grid search.
‫And now we finally got to the end of how you can improve a model via hyper parameter tuning.
‫And again, there's a lot to take in here.
‫The key point is to remember it's an experimental process.
‫Improving a model through hyper parameter tuning is there's no sort of written law on how you do it,
‫but.
‫The point is to just keep trying.
‫See if you can figure out something.
‫See if you can get yourself familiarized with with different hyper parameters that different models
‫have, such as the random forest classifier.
‫And just remind yourself that the first model that you make is more than likely not the best model that
‫you will have.
‫It's a baseline model and it can be improved upon and so built a model.
‫We've tried to improve it.
‫Let's say you've decided you're going to focus on on recall or something like that.
‫You may want to export your model so you can share it or use it in some sort of deployment set up.
‫So in the next section we're going to look at, if we look at our list here, what we're covering,
‫we're going to look at how to save and share and load the model.
‫So save and load a trained model.
‫The benefit of this is that you won't have to retrain a model like we've done in the past few sections.
‫So go back through.
‫Check out what we've covered with hyper parameter tuning how to improve a model hyper parameter with
‫grid search.
‫Try it out with randomized search, try it out by hand and I'll see you in the next video.