‫Wonderful.
‫We've seen how to quickly get a sniff of how our machine learning model is doing and evaluate it using
‫the score method, and that'll return a default evaluation metric depending on the problem we're working
‫on in regression, it's returns to the coefficient of determination and in classification it returns
‫the mean accuracy.
‫However, when you get further into a problem, it's likely you'll want to start using some more powerful
‫metrics to evaluate your model's performance.
‫And so naturally, the next step up from using score is to use a custom scoring parameter.
‫So if we see here, we've looked at this one estimate, a score method, and now we're going to have
‫a look at using a scoring parameter model evaluation tools, using cross validation.
‫All we might have to see what that is.
‫So let's dive into it.
‫Hey, what we're going to do is to work this out.
‫Now, you'll have to bear with me for the next few videos because we're going to cover a fair bit here.
‫The reason being is because evaluating a machine learning model is such an important step.
‫It's one thing to call a fit function on some data, but the next most important thing is going, Hey,
‫is that model actually working?
‫Is it learning something?
‫Could we use that to predict in the future?
‫And so that's why I'm making sure that we're covering a lot of ground here.
‫So let's get started.
‫What we're going to do is go from scale, learn model selection, import cross, vowel score.
‫And as always, we're going to run the code first before we dive into what's actually going on.
‫And what we do need is some classification code.
‫So what we're going to do, we can copy this.
‫You're allowed to copy this, by the way, because we've written it a few times now.
‫So only thing different here of what we've done is we've imported cross vowel score from cyclones,
‫model selection.
‫Now the next thing we're going to do because we've called fit there, we just put a little semicolon
‫so we don't get a big output.
‫You know what?
‫This warning keeps coming up, so we might just keep changing that.
‫An estimate is as equals 100 wonderful.
‫And so cross vowel score, it has the word score in it, but we've seen.
‫What does this cross vowel doing?
‫Well, let's have a look.
‫So go see our score.
‫We're going to do the test data first so we can compare and then we'll do the same.
‫But this time using cross vowel score, cross vowel score and cross vowel score takes our classifier.
‫It takes x data and it takes the Y data, not the test.
‫And not the train.
‫We'll see what's going on in a second.
‫Huh?
‫What's happening here?
‫And again, we're getting another warning.
‫This is just to say that the default value of CVE will change from 3 to 5.
‫So if we have a look at this shift tab.
‫Let's read the docstring.
‫Evaluating a score by cross validation.
‫What even is cross validation?
‫And so CV is going to give us a warning.
‫This is where that warning is coming from.
‫If we change this from CV equals, the default is three.
‫If we change it to five, that warning is going to go away.
‫And now we're going to get an array back of five different scores.
‫Hmm.
‫So that's probably the first difference that you'll notice.
‫Cross vowel score returns an array, whereas score only returns a single number.
‫Okay.
‫So how can we figure this out?
‫We need to figure out what cross-validation is because that's, after all, what cross vowel score is
‫doing.
‫Come back into the docstring.
‫Evaluate a score by cross-validation on Lucky.
‫Here's something I prepared earlier to demonstrate.
‫And as always, I understand things better.
‫Visually, you might do the same to demonstrate what cross-validation is doing.
‫So what we've done before in our normal train and test split.
‫So we split our data into training.
‫So this could be a stay.
‫We started off with 100 patient records.
‫We'd split it into a training split.
‫In our case, we've used 80% and this would contain X train and we train 80 samples and I've used the
‫number 100 here.
‫We've really got more than that.
‫But this is just because the numbers work out visually better here.
‫And in our test dataset, we've got 20% of the data which would contain X test and we test.
‫Now the difference here with cross validation and in our case, this image here is demonstrating five
‫fold cross validation.
‫What you'll probably see cross validation referred to is K fold where K is an arbitrary number.
‫And the reason why we're using five is because if we come back here, CV equals five there.
‫So that stands for cross validation.
‫And this is what I'm talking about, splitting our data into training and test using 20% for the test
‫size.
‫So that means naturally the other 80% goes to the training data, which is where this little split here
‫comes from.
‫Now, what cross validation does is it does five different splits, so it'll use the first 20%.
‫So see how here this is kind of using the last tail 20% here.
‫So it'll create a test data set here and a training data set here.
‫Then it'll move over here and use this as the test data set.
‫And then again and again and again.
‫And so what happens is that cross validation trains five different versions of the model, and then
‫it evaluates that those models trained on each training data on five different versions of the test
‫data.
‫So what's the purpose of this?
‫Well, as you could imagine, if we're only training one model, it could be a lucky split, like,
‫say this, 80% of rows say that had a whole bunch of information and the model was able to learn really
‫well on these 80 rows, on these 80 patient records.
‫And then it got a really good score on this test set.
‫Is that a true reflection of how our model would understand the data or figure out the patterns in the
‫data?
‫Well, not really.
‫Right, because it's just luck, right?
‫If we're just splitting this randomly and somehow a bunch of easy patient records have gotten here and
‫the models figured out certain patterns and it's gone over to here to this test data set.
‫And it's got an amazing score.
‫We could be tricking ourselves.
‫We could be fooling ourselves into thinking that our model is far better than what it actually is.
‫So that's where cross validation comes into play.
‫It aims to provide a solution to not training on all the data and avoiding getting those lucky scores
‫on just a single split of data.
‫So it'll create five different splits.
‫So no matter what, our model is going to be training on all of the data and evaluated on all of the
‫data.
‫And so this is why if we come back.
‫To see cross vowel score.
‫It's why it gives us back five different scores here.
‫Oh, so that's starting to make sense.
‫So if we call the score parameter on only our X Test data and our Y test data, it's going to give back
‫one score.
‫But if we call cross vowel score because we refer back to the graphic here, it's going to make five
‫different splits.
‫And remember, five fold is just an arbitrary number.
‫You could do ten fold, you could do three fold.
‫You could even do 100 fold.
‫But five fold is the default of the library and it's usually pretty good depending on the size of your
‫data.
‫So we'll use five fold to demonstrate here.
‫Just to prove it to you.
‫We can go in here.
‫Cross vowel score.
‫We could even do ten.
‫So this means it's just going to make ten different splits exactly the same as this.
‫And then return ten different scores.
‫So you see here, this is a great example on split one, it's got a score of 0.9, so that could be
‫90%, which is higher here.
‫But on a later split, it's got something much lower, 72.
‫And so what we do here is to figure out a more ideal performance metric or evaluation metric for our
‫model is that we can take the average of these five scores.
‫Let's see it happen.
‫We'll do it all in one cell, random seed.
‫And what we're going to do is get a single training and test split score.
‫I'm going to make sure CLF single score equals CLF score X test.
‫It's going to use the same data here Y test.
‫Wonderful.
‫And then we're going to go take mean of five fold cross-validation score.
‫Clef cross vowel score equals np mean then we need cross vowel score x oh no we need to pass it our
‫classifier x y and we'll use CV equals five and then.
‫Compare the two.
‫So we're going to see a live single.
‫School and then SEAL F Cross Val School.
‫Byrne So what do we see here?
‫Well, in our case.
‫Our original single score, which is now down here, just the exact same number because we're using
‫a random seed using the same test data.
‫Our single score is 0.85.
‫But when we use cross validation, when we use five splits because KV equals five, we get a score of
‫0.82.
‫So it's slightly lower.
‫But in this case, if you are asked to report the accuracy of your model, even though it is lower,
‫you'd prefer the cross validation metric over the non cross validation metric.
‫Now what?
‫We haven't even used the scoring parameter at all.
‫Well, that's because by default, it's set to none.
‫Let's have a look.
‫Scoring parameter set to none.
‫By default.
‫So if we call cross vowel score C, l, f, x, y, CV, we can pass scoring here and it's going to
‫be set to none.
‫So if we do shift tab on this, how do I know this?
‫Well, luckily the docstring comes in handy.
‫C By default it's set to none.
‫So if we keep scrolling down.
‫Move our notebook so scoring string callable or none optional default none are OC a string C model evaluation
‫documentation.
‫That's what we've had a look up here.
‫Or a score a callable object slash function with signature score estimated x y which should return only
‫a single value if none the estimate is default scorer if available is used.
‫R Okay, now this is why we know that this is accuracy, because if the scoring parameter of Crosswell
‫score is none, it uses the default scoring parameter of our estimate in an owl case.
‫What is it?
‫The default
‫scoring parameter of classifier equals mean accuracy.
‫And where did we see that before we saw that in the last video?
‫When we go see a live score and we hit shift tab returns the main accuracy on the given test data and
‫label.
‫So that means when we have scoring set to none, it's going to use the default evaluation metric for
‫cross-validation on our classifier.
‫So if we hit shift and enter.
‫It's going to return the same values.
‫Or it might be slightly different, right?
‫Because we haven't set up a seed in this cell.
‫So these values are going to be different to the cross cell score we see up there.
‫If we'd run it in here, we would have seen similar values.
‫What have we covered here?
‫Well, as you might have guessed, the scoring parameter can be changed.
‫Right?
‫So we can, as the docstring says, we can import our own scoring parameter here.
‫We can change this to something other than none.
‫That is what we're going to start to cover, right?
‫We're going to have a look at in the next few videos, some other classification model evaluation metrics
‫that we can use with cross validation.
‫And so while we use cross validation, well, as we saw.
‫In the picture.
‫Cross-Validation aims to solve the problem of not training on all the data.
‫Right.
‫So we're creating five models and we end up having a model trained on all of the data and avoiding getting
‫lucky scores.
‫So training on a single split.
‫And we saw that in action.
‫With our single split score here getting a slightly higher score than our cross validation score.
‫Whoo!
‫That's a lot.
‫Now, we've still got a bit more to go.
‫Let's get some in-depth classification metrics happening.
‫I'll see you in the next video.