‫We've got our data now in X train y, train x validation, y validation.
‫But in a previous video we evaluated our model using the score function.
‫What does a school function use?
‫It uses the coefficient of determination, but if we remember right back up the top, we'll go right
‫back out at the top.
‫Just to check it out, because we wrote this down.
‫We know that the evaluation metric for this competition, so for this project that we're working on
‫is are immensely or in other words, root means squared log error.
‫That's not the same as coefficient of determination.
‫And if you're wondering where we got this evaluation function from, it's from the competition overview
‫evaluation, which we saw right at the start, but we'll just check that out again.
‫Evaluation.
‫So the evaluation metric for this competition is the arm's length.
‫We can Google that to see what it actually is.
‫Let's go here.
‫Root mean squared log error.
‫What is the difference between how do you interpret root mean squared logarithmic error?
‫Okay, so we've got a fair few options there.
‫So if you wanted to read further, these are something you can look up.
‫But we're going to work through this in code.
‫Let's go down.
‫It might be wondering, we haven't seen all really mentioned root mean squared log error.
‫That's a bit of a mouthful in itself.
‫So what we might do, we're going to have to create our own evaluation function because if we come back
‫to our keynote, so classification and regression metrics, we're working on regression problem here.
‫This is one of the slides from I think there's a cyclone section.
‫We've got a main, absolute error OC mean squared error, MSI, OC root mean squared error.
‫But this is missing the log component.
‫Let's see if we could get that in through src cyclone.
‫So we'll go back here.
‫What we might do is build our own.
‫So building and an evaluation function.
‫Now this is something you may have to do depending on the project you're working on, right?
‫So if we're working on this cable competition and they say that for some reason we have to use this
‫metric, who knows?
‫The people who posted this challenge may require this.
‫They may have decided that this is their best evaluation metric.
‫Depending on what project you're working on, you may have to find an evaluation function which suits
‫your use case the best.
‫And so what we're going to do is create a valuation function because we're going to be running multiple
‫experiments.
‫We want to be fitting multiple different machine learning models and evaluating them.
‫That's why we're going to build it into a function so we can use this functionality multiple times without
‫having to repeat ourselves because we're kind of lazy when it comes to writing all this code, you know?
‫I just want to experiment more and more and create a valuation function.
‫The competition uses root mean squared log error SLA so we can go from scalar and matrix.
‫And I can tell you psychic learn, you could go to the SRC, learn evaluation matrix, but we can go
‫through here, we can have a look at some evaluation metrics, mean squared, log error.
‫Okay.
‫So we might use that.
‫Main squared log era, but we're going to have to add our own little piece on it because this is not
‫root mean squared log error.
‫So what we might do is we might take this one.
‫We can import that.
‫Let's do that.
‫So from SQL Learned Metrics, import main squared log error and for the hell of it will do mean absolute
‫error.
‫You make predictions on all the examples and then you minus your predictions from your actual values
‫and take the average.
‫That's the mean absolute error.
‫So how much on average is your prediction from the actual sale price?
‫But main squared log error is more to do with the ratio and most often times with regression problems,
‫right.
‫If you like $10 off something like.
‫So if we were $10 off this sale price prediction wouldn't really matter.
‫But if we were 10% off, that'd be pretty bad, right?
‫You don't want to be more than 10% off.
‫Whatever value insert something.
‫It could be 20%.
‫It could be 30%.
‫But that's kind of the difference between these two, right?
‫Any time you see absolute think of it as being $10 off and any time you see like squared or some sort
‫of log or think of it like the ratio.
‫So being like 10% off, that's when these two you can really compare them to each other.
‫But we've got a little bit more of that in the slide.
‫You might want to take a screenshot of that.
‫That is a pretty good one.
‫But this will all be in the slides you can check out later.
‫We want to create an evaluation function.
‫We've got main squared log error.
‫So how would we turn that into root mean squared log error.
‫So let's try it out.
‫So maybe we go here.
‫I don't want to type out root mean.
‫Nah, that's too much.
‫Ah se mess l e root mean.
‫Squared log error.
‫Beautiful.
‫And that can just take why test and why it's going to take the same inputs as my psychic learn function.
‫So see how this takes true why read it can be y true it can be why test doesn't really matter why test
‫and then we go here we'll leave a little docstring for ourselves.
‫Calculates
‫rate main squared log error.
‫Between predictions and true, true labels.
‫Wonderful.
‫And then we might go return.
‫Just nice and simple.
‫We can just use square root of mean squared log error.
‫We'll have to pass this to the test values and the predictions to mean squared log error.
‫That's actually pretty easy.
‫What we're doing here, we're taking the square root.
‫So, right.
‫So the root component of the main squared log error, the main squared log error between predictions
‫and true labels.
‫Wonderful.
‫So we've created our own evaluation function that lines up with Cargill's requirements, the root mean
‫squared log error.
‫And we probably want to check a few more things like we want to create an evaluation function to check
‫a few things at the same time so we don't have to keep calling something like this.
‫So let's create a function to evaluate model on a few different levels.
‫So again, evaluation is just as important as building the actual model.
‫So we'll go here, show the scores model, so we'll pass it our model.
‫We'll make some predictions on the training data, some model predict.
‫So we'll evaluate how our model did on the training data X train and then we'll evaluate how our model
‫did on the validation data model DOT predict, we'll pass it out X validation here.
‫If our model is performing better on the validation data set, that's hinting at us that our model is
‫overfitting.
‫So between training and validation generally, because our validation set only has 12,000 examples and
‫we're not training our model on it, you'd want to see slightly worse metrics on the validation set
‫than versioning the training set.
‫So we'll go scores equal.
‫Maybe we'll return a dictionary of some sort.
‫Yeah.
‫So training may a.k.a mean absolute error.
‫We're going to go.
‫Can I tab.
‫What would I complete mean.
‫Absolute.
‫Yes I can y train.
‫So remember we're just comparing the y train.
‫So this is the training mean absolute error on the training labels versus the training predictions and
‫beautiful we're going to go valid may so again you'd expect this valid MA to be slightly worse and the
‫training this is going to be y valid and this is going to be vowel breads and then we're going to go
‫training muscle for root mean squared error and we can do this using our fancy little function that
‫we've just created up here.
‫This is exciting.
‫We're writing our own evaluation functions.
‫Maybe that's a pull request potential to psychic lion.
‫Wonder why they haven't got root mean squared log error.
‫Maybe it's not that popular, or maybe you can just easily create it yourself like that.
‫Is that correct?
‫Why?
‫Train, train heads?
‫I'm talking too much.
‫Don't talk in code, kids.
‫No, I'm kidding.
‫You can do whatever you want.
‫Here we go.
‫Here, MSL, except don't under evaluate your machine learning models.
‫That's the only thing you're not allowed to do.
‫Everyone loves a good model, but everyone loves a well evaluated machine learning model even more.
‫And finally, four for good measure.
‫Since we're here, let's do the default score and oh, maybe we just import that.
‫That's probably a bit better too.
‫Is it to score?
‫What is this?
‫Why is this not working?
‫Tab What I complete is giving me some issues.
‫So we've got to arch just on a R2 score.
‫Yeah.
‫Trust your instincts and you'll come on to score.
‫So that's all we're changing there.
‫Rather than re something, we'll just import it.
‫Now, this is fairly decent function.
‫Not going to lie.
‫Takes care of a few things.
‫Try and Fred's probably some redundancy we could fix.
‫But that's alright.
‫We're trying to experiment as fast as possible.
‫You might be thinking, wow, our experiments are getting slowed down, but with your own practice,
‫without having to listen to me talk and go through these sort of things, you'll be whizzing through
‫experiments, you know that's wrong, isn't it?
‫This needs to be vowel parade's, vowel prince.
‫Beautiful.
‫And we're going to return the scores.
‫Okay, so what have we done here?
‫So we've built an evaluation function in line with what's required for this particular project, and
‫in our case, its root mean squared log error.
‫I'm going to go hoarse saying that if I keep doing it and then we've created another little helper function
‫that's going to do a whole bunch of other metrics.
‫It's going to make some predictions using a model we pass it to, and then it's going to compare those
‫predictions on a whole bunch of different playing fields.
‫On the mean absolute error on the root mean squared log error and on the coefficient of determination.
‫Whew.
‫So that's probably enough for this one video building our own custom evaluation functions.
‫Now, again, this is something you might want to do for your own projects because we're running as
‫many experiments as possible.
‫It is helpful to have evaluation functions like this so you can quickly do a whole bunch of different
‫things rather than having to retype out this every time we try a new model.
‫But.
‫Let's press shift and into Matt.
‫So our functions are instantiated and we're going to try and test them out the next video.