‫Oh, right.
‫So now we've got our own custom evaluation functions.
‫We can evaluate how our model is doing.
‫So the next logical step would be to train another one on the training data set and then evaluate it
‫on the validation data set, or just using this beautiful function that we've crafted.
‫Not so fast.
‫Let's see why.
‫So testing our model, this section is going to be we're going to do it on a subset.
‫So testing our model on a subset.
‫Now this is mostly to tune the hyper parameters.
‫Because remember what we've been stressing, we want to decrease the amount of time it takes us between
‫experiments and if we come back up, remember how we time traveled in a previous video and that took
‫6 minutes, 58 seconds to fit a model on all those data.
‫So on 400,000 rows, we don't have enough time to do that again.
‫We can only time travel every so often.
‫And so if we were to just ride it out again, if we would instantiate our model.
‫So model equals random forest regress, other end jobs equals negative one and a random state of 40
‫to wonderful.
‫And then if we just did model to fit x train, I need to be careful here not to be trigger happy if
‫we did this.
‫Right if we put a little time up here.
‫Time.
‫That's going to take far too long.
‫All right.
‫So I might put a note here.
‫This takes far too long for experimenting, because that's going to take you about 5 minutes.
‫That's not what we're after.
‫And again, if you have run this cell before, actually, I probably should have told you this earlier.
‫If you did run this cell on your computer and it's taking far too long for it to load, remember, you
‫can always press the stop button up here.
‫So if we run this.
‫Oh, well, that's too fast.
‫That runs too fast.
‫But if your cell is running, remember how there's a little star here?
‫I'll show you.
‫There'll be a star.
‫Well, that that happens too quick for this is you're going to have to trust me on this one.
‫When you run a cell, this will turn into a star.
‫If it's running, you can stop it by pressing this button with that cell highlighted.
‫And that will stop that cell.
‫So we come back.
‫Let's not run that.
‫We'll make sure we can't.
‫So we comment it out.
‫Beautiful.
‫Because why?
‫Because our dataframe is 400,000 rows there.
‫And my little MacBook Pro has already calculated a lot of patterns in 400,000 rows.
‫And it's like, well, I'm going to take a while, so you need to be patient with me.
‫Well, that's fair enough, because I would take far longer than my MacBook Pro does to calculate three
‫400,000 rows.
‫So what our solution is, is to speed up our time between experiments and that's what we're doing,
‫testing our model on a subset.
‫We could do something like this, could just copy this model, not fit.
‫Maybe we want to only do it on 10,000 rows.
‫But there's an even better way.
‫And it's right built into.
‫We could do that.
‫Right.
‫That's going to test it on 10,000 rows.
‫But now there's a better way to do it.
‫That is one option.
‫So keep that in mind that you can just slice your your training data set.
‫But the beautiful thing about random forests and this is why we're focusing on it so much, is that
‫they have this little feature in it called Max Samples.
‫So change max samples value.
‫All right.
‫And if we wanted to figure out what Max Samples does, where would you go?
‫Well, one option is if we go model equals rainforest regress, we're going to have heaps of practice
‫instantiating these models.
‫It's what it's about, though, Ryan.
‫It's just about almost making this second nature of what it's like to just get a model, instantiate
‫it and get it ready.
‫Random state equals 42.
‫There's our model.
‫So that could be one option.
‫But if we press shift tab here, we'll see the docstring max features not max samples.
‫Max samples.
‫Boom.
‫And if we come down here, let's find it.
‫Max samples.
‫There we go.
‫If bootstrap is true, the number of samples to draw from X to train each base estimator.
‫Hmm.
‫What does this mean?
‫So if we go model.
‫Okay.
‫Bootstrap equals true?
‫Yes.
‫And now each estimate a c's max samples.
‫Now, where's Max samples?
‫None.
‫So what does it mean if none?
‫It's going to be all, isn't it?
‫Let's have a look.
‫This is one I want you to have some practice doing right is if you're not sure on something, it's looking
‫it up and reading it in the docs.
‫If bootstrap is true, the number of samples to draw from x so x is what we pass the model.
‫If none, then draw x shape samples, so it's going to draw them all.
‫Now if ent then draw mac sample samples.
‫So.
‫What this means is that if we go x train dot shape zero.
‫So if we leave max samples as none.
‫Every estimate.
‫In our case, there's 100 because the default estimate is in random forest, aggressor is 100.
‫So you can imagine an estimate as a small model.
‫So each small model, so 100 small models are going to see every single one of these.
‫So that's that's a large number.
‫Right.
‫So this is all the calculations that we have to make probably more than this, but this is what you
‫can imagine.
‫But instead, if we set max samples to something else, so say for example, maybe we only wanted to
‫to try 10,000 examples instead of 400,000.
‫Right?
‫So closer to the shape of our validation set, we could change max samples set to 10,000.
‫So this means that instead of looking at 400,000 rows, every estimator.
‫So the 100 estimates for an estimate is we could also change this, to be honest, but we'll leave it
‫as default just for this example.
‫So that means because we change it to 10,000 times 100.
‫Well.
‫So let's have a look here.
‫Let's divide these two.
‫How many times?
‫Less.
‫What do we got here?
‫What number is that?
‫How many zeros is there?
‫A million.
‫That's a million.
‫I can count.
‫40 times less so 40 times less data to compute on.
‫So that means we should see a speed increase of 40 X or about that, right?
‫Because it's not going to be perfect.
‫So let's go here.
‫Change what we might do.
‫Yeah, that looks good.
‫And then we'll create a new cell here.
‫We'll leave our little time here because this is what we want to practice.
‫So cutting down on the max number of samples each estimator can see improves.
‫This is our hypothesis improves training time and let's have a look model fit.
‫We'll do it on our training data now.
‫So this shouldn't take too long.
‫Maybe we might even have time to wait for it.
‫Still might take a few seconds.
‫This might be another opportunity to time travel.
‫Hall.
‫I didn't need its own travel much.
‫Look at that whole time.
‫16 seconds instead of 7 minutes.
‫Now, that's a bit better, right?
‫Now, that's something that we can potentially just sit here and wait for.
‫If you're doing seven minute experiments all the time or even longer, if you had more data, you're
‫going to lose patience, you're going to get distracted.
‫But if we can keep our experiments like somewhere around 10 seconds, so that's that's pretty good,
‫right?
‫I can wait 10 seconds.
‫And now the beauty comes in here when we can call our show score's function.
‫So let's see what happens when we go show scores.
‫Hopefully this works because we've coded up this function ourselves.
‫Of course it will show scores.
‫What are we going to find?
‫Oh, no.
‫We've messed it up by has no spirit.
‫Instead of square root.
‫This is where we've messed up num pi.
‫There we go.
‫Tebow classic couldn't get it right the first go.
‫Could you, Daniel?
‫Nope.
‫So we'll see how our model does.
‫We've just trained it here on a subset of the data.
‫Oh, beautiful.
‫Oh, right.
‫Look at that.
‫So training mean absolute error.
‫So valid.
‫Yeah.
‫See here how I said the valid error will be slightly higher than the training.
‫That's good.
‫That means we're not overfitting and our squared off for training and valid or far worse than the 0.987
‫we saw before.
‫But that's alright because we're only training on a subset of 10,000 examples, not all 400,000.
‫So we'd expect these metrics to be worse than if we were to set max samples to be none or train on all
‫of the training data.
‫Now what we might do is see this is how valid are mossley of 0.29.
‫I wonder where if we were to compare that figure with the leaderboard, where would we end up?
‫So point these are.
‫There's about 425 teams in this competition.
‫So if we were to submit our sample as it is now with our valid arm sleep score of 0.293, only on 10,000
‫samples, where do we end up?
‫0.293.
‫Wow.
‫We're in the top 100.
‫Look at that.
‫We're in the top 100 with just a simple model on 10,000 examples.
‫So somewhere in here.
‫All right.
‫Now we've got an idea of how to train our model a bit faster.
‫What we might do is try some different hyper parameters, and I'll let you think about it before we
‫get into the next video.
‫What's a way to find different hyper parameters rather than us having to manually adjust to these hyper
‫parameters of our random forest or aggressor?
‫If you're not sure, don't worry.
‫We'll cover it in the next video, but have a think.
‫How have we found better hyper parameters in the past?
‫Because that's what we're going to use in the next video.