‫Now we've got an ideal model that is performing pretty -- well on the validation data set.
‫So valid arm's length is the metric we've been paying attention to because that is the evaluation metric
‫for this particular Kaggle competition.
‫Of course, if you're working on a different problem or a different set of data, you may have a different
‫evaluation metric that you're trying to improve.
‫But in our case, we're doing pretty well.
‫So let's use our ideal model to make predictions on test data.
‫There's a little tidbit in this section that's really important to remember, and we're going to come
‫across it in a second.
‫So first of all, we're going to have to import the test data.
‫Now, if we have a look.
‫In our data folder.
‫So this is just in the Jupiter home because we've created a project folder for this particular project
‫and we've downloaded our data, put it in here.
‫Inside here there's a data set which is test CSV.
‫So this is what we're going to be using.
‫And if we come back to Kaggle, it tells us test CSV is the test set which won't be released until the
‫last week of competition.
‫What we've got at the moment, since this competition has already passed, it contains data from May
‫one, 2012 to November 2012.
‫Okay, so it's about the six months or so after our validation set.
‫Beautiful.
‫And our models performing okay on the validation set.
‫So what you might see is see how we can make predictions on this future six months of data.
‫So we'll go rdv test equals PDF or read see as v.
‫We're just going to do it the exact same we did at the beginning with our train and valid data set.
‫Blue Book for bulldozers.
‫Wonderful.
‫And it's called test csv and we'll pass in low memory equals false.
‫And we're also going to pass dates because remember we haven't done any manipulation to this data set
‫like we have to our other ones.
‫We'll see where that comes in in a moment and then we'll check DF test dot head.
‫Oh.
‫What's happening?
‫Oh, typos.
‫Beautiful.
‫So this looks familiar.
‫It's something that we've seen before.
‫Except now.
‫This time, the test data set is missing the sale price column.
‫And the reason being is because that's the column we're trying to predict.
‫So why don't we see if we can do that?
‫So we go test spreads equals ideal model.
‫We use our ideal model this time model predict the ideal model has been performing the best and we should
‫be able to just go pass it the whole data frame, right?
‫Because this will be our X values and we don't need to drop any columns because it doesn't have a sale
‫price column.
‫So let's see what happens.
‫Actually, they put a little comment here, make predictions on the test data set and I just thought
‫of a little challenge before we run this cell, I want you to kind of think about what might happen.
‫So our machine learning model, it's been trained on a particular data set and now we've imported the
‫test data set.
‫What's going to happen if we try to make predictions on this data set based on I'll give you a hint
‫based on the data that our model has been trained on.
‫And if you're not sure, that's fine.
‫We're going to see it in a sec.
‫We should see it right now.
‫Oh, there we go.
‫Value error.
‫Okay.
‫What's happening here?
‫Could not convert string to float.
‫Oh, okay.
‫So if you remember right back up the top, let's scroll back up through our beast of a notebook that
‫we've been working through.
‫What did we do before we could train a machine learning model?
‫We saw that a similar error when we first started right up here when we started modeling.
‫It'd be great if I could find it.
‫Our model.
‫Didn't work.
‫Here we go.
‫Okay.
‫Build a model.
‫So DF template drop.
‫Could not convert string to flight.
‫So the same error message is what's below.
‫Now the reason being this has happened is what do you think?
‫Well, it's because when we imported our original data set, we did some manipulations to it.
‫What did we do?
‫We made sure our data was numeric and we made sure there were no missing values.
‫So what we've done now is we've imported DF test.
‫It might have some missing values, which it does.
‫And it might not be all numeric.
‫And.
‫Far out.
‫We've done a fair bit of data processing here.
‫And if we take the columns.
‫So these are all the features.
‫But if we check out ex train columns.
‫Oh, don't need brackets there.
‫We noticed that the column lengths are different.
‫Hmm.
‫So there's 102 columns in our x train.
‫Whereas in DF test, what have we got here?
‫We saw it up here before when we did dot head.
‫52 columns.
‫Hmm.
‫So the reason why our machine learning model can't predict on the test data set at the moment is because
‫it's not in the same format as the model was trained on or as the data set the model was trained on.
‫So that's what we're going to have to do.
‫All right.
‫Let's get rid of these here.
‫And so what?
‫How can we do this?
‫So we might go pre-processing the data.
‫In other words, getting the test data set in the same format as our training data set.
‫Wonderful.
‫So how might we do this?
‫So what we might do, I think, is create a function for doing so.
‫So if we were to import a new data set, say a test data set, we actually did some manipulations to
‫our training data set.
‫We could take those functions that we did on the training data set and just do the exact same thing
‫on the test data set.
‫So that way they're both the same.
‫That makes sense.
‫Let's see it first.
‫Let's see if we can do this.
‫So pre process data or pass it a data frame.
‫We'll leave a little note here.
‫So performs transformations on DF and returns transformed DF simple, right?
‫We want to create a function which takes a data frame of some sort.
‫It does a bunch of manipulations on that data frame, and then eventually it's going to return that
‫data frame.
‫So now we're going to have to fill in all the transformations that we did originally to do temp in here.
‫And so we can do that by going up.
‫We might have to do a few copies and pace here.
‫So remember right back up the top.
‫So this is a little rule, a little tidbit.
‫What you do to the training data set, you're going to have to do to the test data set.
‫So that's a that's a machine learning tidbit there.
‫Okay.
‫So here's the first big set of manipulations that we did.
‫It might copy that.
‫Wonderful.
‫And we can just remember this line.
‫So we've we've added some information here based on the date time column sale date, and then we removed
‫it.
‫So they're the first two steps that we have to do.
‫Okay.
‫We'll scroll right down to the end of our notebook.
‫It's going to be a lot of scrolling in this video because we did three major things right.
‫We added some information here to the date time or we extracted.
‫That's probably a better idea.
‫We extracted information from our date time column and added it to the data frame.
‫So we might go def dot drop and after we've done this, we'll drop the sale date column.
‫And we'll go axis equals one and in place equals true.
‫Wonderful.
‫And we have to remove temp from all of these because we're only using DF.
‫So a little trick here is by moving your cursor and holding down command if you're on a mac, or maybe
‫it's control if you're on windows and we can just go backspace here.
‫Wonderful, right.
‫Because we're not working with DF temp, we're working with DF.
‫So any arbitrarily named DataFrame will work with our function.
‫So the next thing we did was we need to fill the numeric rows with median.
‫And we also.
‫Filled categorical.
‫Missing data and turned into numbers.
‫And turn categories into numbers.
‫So that's what we have to scroll back up and find.
‫Let's let's do that first.
‫We'll get the missing missing new miracles.
‫Luckily, we've laid our notebook out.
‫We've added these headings here so that we can see what we were doing.
‫This is all part of helping yourself out, right?
‫So fill numeric rows with the median.
‫We actually need all of this code here.
‫We'll copy that.
‫Beautiful.
‫So this is where comments and different headings in your notebook can really help your future self out
‫is when you want to function eyes things and clean things up.
‫So there we go again.
‫We have to remove DF temp so we get DF temp.
‫We don't want that DF temp, we only just want DF.
‫Okay.
‫Beautiful.
‫And we still need to fill categorical missing data and turn categories into numbers.
‫Now, let's just see if we can remember how to do this rather than having to scroll back up, because
‫we've done enough scrolling, to be honest.
‫So we can do this.
‫If not, if we remember, if it's not a numeric type, that's how we did it.
‫Typekit is numeric type content because we're looping.
‫We're in this same loop here.
‫Content we want to go.
‫Def label plus underscore is missing yes equals.
‫So it's going to check if it's null, if it's null.
‫We want to add a label that a label is missing, that that particular data point is missing.
‫And remember how we add plus one to the category code because pandas and codes missing categories as
‫negative one.
‫So this is where we can go.
‫Def label equals PD categorical, so turn it into a categorical.
‫So turn the content of a particular column into a category and then access its code aka turning it into
‫a number and then plus one.
‫All right.
‫Not a bad function.
‫So this is what kind of happens in a notebook workflow, right?
‫Is that up here?
‫You could probably classify most of this as pretty messy.
‫We've helped ourselves out with different headings and whatnot in comments.
‫However, as you sort of get further into the process, you'll start to build these helper functions.
‫So you're collating a whole bunch of different steps into a function so that it really syncs things
‫in the notebook.
‫And you can just call a function like this, much like our show scores function, and it does a process
‫that's going to be the same every time, rather than having to scroll up through the notebook and find
‫something in a different cell.
‫So now we can hit shift enter on that or what's happened expected missing block.
‫I didn't think we needed one.
‫Oh.
‫That's why we need it.
‫We need it in dance here.
‫There we go.
‫Classic.
‫This is what happens when we copy and paste code.
‫Right?
‫The formatting isn't always perfect.
‫Beautiful.
‫So we have a function here that's going to pre process some sort of dataframe.
‫It's going to add some information based on the sale date column.
‫It's going to fill numeric rows with the median.
‫It's going to turn the missing categorical data into a zero value and the rest of the data into then
‫category code number.
‫Beautiful.
‫So it's going through a few steps.
‫But that's alright, we've functional as it's all going to happen together.
‫And now what might we do here?
‫Actually, a good question is where might this function break.
‫I might leave you on that for this video to save it from getting too long.
‫I want you to have a look at this big function that we've created and think about.
‫If we passed it, our training data frame, how might it work?
‫And then if we passed it, our test data frame, right DF test, which is up here, we passed it this
‫data frame.
‫How might it break?
‫We'll answer that in the next video.