‫In the last video we left off scoring our machine learning model that we fitted to the data after we
‫just filled all the numerical values, filled all the categorical values, and turned all our data into
‫numbers.
‫Model worked.
‫Our data had no missing values and it was all numeric.
‫But we left off with the question Why isn't this metric reliable?
‫Let's put that here, because that's worth highlighting, because remember, always remember fitting
‫a machine learning model or evaluating a machine learning model is just as important as fitting one.
‫And that's where we're up to now because we've just fit a machine learning model.
‫We have to make sure that our results hold water, right, because we don't want to be promising things
‫that our model can't do.
‫So question why doesn't the above metric hold water?
‫Hold water is an expression for why isn't the metric reliable?
‫That's just a way of speaking.
‫So why doesn't it hold water?
‫Why isn't it reliable?
‫Have you had to think about it?
‫I'll show you a diagram that may refresh your memory if we come back to our keynote.
‫You remember from right back at the beginning, the most important concept in machine learning.
‫Okay, the three sets.
‫So what we've done is we've trained a machine learning model on one data set, but then what have we
‫done?
‫We come back to the code.
‫We fit the model.
‫On the data set on some data, but then we've evaluated it on the exact same data.
‫Uh.
‫Och, so if we come back here, essentially what we've done is if we were thinking about this as being
‫a university course, we've learned the course materials.
‫But what we've done, instead of testing our model on the equivalent of like a final exam, aka a test
‫data set, we've just evaluated our machine learning model on the exact same materials that it's learned
‫on.
‫You were in one class and you got given a book to read, and then at the exact same time, it's just
‫like you're going to be tested with the questions directly from that book you're reading.
‫Hmm.
‫Okay.
‫But we after our model's ability to generalize and acquire the ability for a machine learning model
‫to perform well on data it hasn't seen before, that's what we're after.
‫If we go back to Kaggle.
‫There's a training data set which contains data through the end of 2011.
‫There's a validation set and there's a test set.
‫So that's what we're after.
‫We need to evaluate our model, not on the data we've trained it on, which is this we need to evaluate
‫it on the test data.
‫But before we do that.
‫If you remember right back up at the top, let's go right back up the top to we imported our first data
‫frame.
‫I said we'd revisit this and now the time has come.
‫So we go here.
‫We imported it.
‫Train and valid csv.
‫Hmm.
‫Wonder why did we do that?
‫Well, it's all going to become clear in this video.
‫So we've got train invalid csv, we have a valid csv and we've got try and see as v, why don't we just
‫import them separately?
‫Well, I wanted to demonstrate what it's like creating your own validation set rather than someone else
‫creating it for you with a time series data set, which is what we're working on.
‫So this is a perfect playground for that.
‫So if we read here train CSV is the training set.
‫So basically what we've been working on and valid CSV is a validation set.
‫The key point here is because it's a time series, the training data set has data up to the end of 2011,
‫whereas the validation set has data between January one, 2012 to April 30, 2012.
‫So what we will do to better evaluate our model, we need to split our data up into training and validation
‫set.
‫So let's do that.
‫Let's go here splitting data into train validation sets because that's what we're after.
‫Now we've got a model.
‫We can build models now, but we need to rather than just build better models, we need to make sure
‫that what we're evaluating with makes sense.
‫So let's check our data again, as we always do.
‫So maybe we can use the year, what was it called, say.
‫Oh yeah.
‫So do you have to say.
‫Oh, yeah.
‫There we go.
‫Okay.
‫So maybe because our dataframe is in order.
‫Yes.
‫This is going to help us when we import our data frame.
‫We ordered it in the order of the sale dates.
‫So now reading this to create our own validation set, we want to split our data.
‫So all of the rows up to 2011, so the sale year is 2011 that can be in the training set.
‫And then all of the rows in 2012 can be the validation set.
‫We won't worry about the test set for now because that's in a separate data set.
‫The reason why we're only working on train and valid CSV is because those we imported the train and
‫valid CSV at the start of the notebook.
‫This file here.
‫So we have to create our own validation data set.
‫So let's do that.
‫So we've got dev templates, check the sale year value counts.
‫There we go, 2012.
‫So there's 11,573 samples in 2012.
‫All right.
‫So to split our data into training and validation, it should be as easy as going, okay, let's introspect
‫this.
‫Our year column and every column that's equal to 2012 will be a validation set for sale year and example
‫and every row that's not equal of so year, that's not equal to 2012 will be in the training set.
‫So let's stop talking about it, Daniel, and let's see the code.
‫So split data into training and validation.
‫This is something that you might have to do with your own time series data, right?
‫Because you're not always going to be given in a Kaggle format, right?
‫When you're working with a client or working on a project, you're not always going to automatically
‫have your data in train, valid and test.
‫These are things you're going to have to create for yourself.
‫Looking at this sale column or looking at the time column, looking at the date column, and figuring
‫out how you can make your own training and validation set.
‫So that is exactly why we imported them as one set to begin with, so we could practice making our own
‫training and validation sets.
‫And I feel like I'm saying the word set a lot, but that's important because we go back to our.
‫Keynote.
‫This is the most important concept in machine learning because whatever we train our model on, we want
‫to make sure we're evaluating it on something else.
‫So come here to the validation set is every row in DF temp where the sale year column equals 2012?
‫Yes, that's correct.
‫And the the day of training set is every row in DF temp where the same year column is not equal to 2012.
‫Wonderful.
‫And then we might go, Len.
‫DF Val and Len df train.
‫So all this is going to tell us is the length of these two data frames.
‫Beautiful.
‫So now we have a validation set which contains 11,573 rows and a training data set which contains 401,125
‫rows or samples and their split on date.
‫Beautiful.
‫We're taking boxes here.
‫We are taking boxes here.
‫So now what we might do is split data into X and Y.
‫We've seen this before.
‫And so that way we have an X training set and a Y training set and an X valid set and a Y valid set,
‫which is our data and labels.
‫So we'll have X train, y train equals DF.
‫Now we're working with the training set here drop.
‫We want to drop the sale price column on the first axis a.k.a just drop the column and we're going to
‫set it to DF train.
‫Y train is going to be equal to sale price.
‫So see how I'm doing it with a comma here.
‫I'm being a bit tricky by just doing it in one line.
‫Beautiful.
‫And we're going here.
‫Valid y valid.
‫Not sure am df output drop sale price just the same again, but this time with the validation set df
‫val dot this is actually going to be sale price.
‫There we go.
‫And now we'll have a little gaze inspect our data just to make sure we haven't made a little error somewhere
‫that you should all be comparable shapes to each other.
‫Why?
‫Valid shape.
‫So we're just taking these data sets that we're creating here and we're finding out the shapes of them.
‫Beautiful.
‫So our training set is 401,000 rows or about that with 102 features, 102 columns.
‫Our Y training is 401,000 rows and then validation is about 11 and a half thousand with the same amount
‫of columns for X and no columns for Y because y is just one column.
‫Let's have a look.
‫Now, look, this is what we do.
‫Make sure all of our data.
‫Okay, so these are all sale price.
‫Beautiful.
‫Woo.
‫Okay, so we've got our data into train and validation sets.
‫We've taken care of two of the three most important things or most important sets.
‫We'll have a look at the test set later.
‫But this is what we've created, a training set and a validation set.
‫It's time to keep building some more models.
‫So we might end this video here and we've got to figure out a way to evaluate our machine learning model.
‫So we'll probably have a look at that in the next video.