‫Okay.
‫So now we've got this behemoth of a function that's going to pre process our data.
‫Let's try run our DF test that we just imported up here and try to make some predictions on.
‫So we imported it here.
‫Tried to make some predictions on it, but it didn't work because it's not the same format as the training
‫data.
‫Let's run it through our pre process data and hopefully.
‫If we've done it correctly, it's going to make the same changes on DF test as we have on the training
‫data.
‫So let's go here.
‫DF test equals are actually we'll put a little common here process test data and we might just re instantiate
‫to make sure that we've.
‫We'll hit shift, enter here, shift into here, just to make sure that these cells have run.
‫I've been notorious in writing a function and then not running the cell so it actually doesn't work.
‫Pretty process data df test and then we should be able to check it out by going df test head.
‫OC 101 columns.
‫Oh, that's looking better.
‫So we're up.
‫We're up a fair few columns from what we started with.
‫We go back here and we started with 52 columns.
‫Now, if we go through here.
‫Okay.
‫On the end, there's a fair few missing.
‫All right.
‫Okay.
‫So it's going to be missing the sales price column.
‫Okay, that makes sense.
‫But still a different shape to X train.
‫So we've got 101 versus 102 columns.
‫And now, if you remember, in the end of last video, I asked to have a think about where this function
‫might break.
‫And if you're not sure, that's fine.
‫I've stumbled across this many of times, right in the process of trying to make our training data or
‫our test data in the same format as our training data.
‫And what it is, is that the test data, when we imported it, is slightly different to the training
‫data, as in let me demonstrate we're doing model driven EDA, so we'll try to make a prediction again.
‫So make predictions on updated test data.
‫We go test spreads equal ideal model predict DF test.
‫We get value ever again.
‫Hold on.
‫I just processed it.
‫What's wrong?
‫Let's check it out.
‫Number of features of the model must match the import model and features is 102.
‫Okay, that's what our model has been trained on.
‫And input features and 101.
‫Hmm.
‫Well, we just saw this when we went x train dot head.
‫That's 102 columns.
‫So there is a difference in the number of columns that our test data frame and our training data frame
‫have.
‫Hence why our machine learning model doesn't work.
‫So how would we figure out where the difference is?
‫Well, we can do that using the columns attribute and turning them into sets.
‫Let's have a look.
‫So we can find how the columns differ using a python set.
‫If you've heard of sets, they're basically a list with only unique values in them.
‫So we can go set X train columns
‫minus set DF test columns.
‫And now this is going to tell us which columns are incorrect.
‫Are auctioneer's ideas missing.
‫So our DF test has no column.
‫AUCTIONEER ID is missing.
‫So what that means is when we imported our test data frame according to our function, it didn't have
‫any auction ID values that are missing.
‫So what we can do is manually update a column on our DF test data frame to have all false values for
‫auction ID is missing because the reason being there is no column auctioneer ID then this is having.
‫If you can't tell I'm having trouble saying auctioneer ID the reason why our DF test dataframe has no
‫column named this right this this here.
‫What I'm highlighting is because all of the auctioneer ID values were filled in DF test.
‫So what we have to do is manually adjust DF test to have auctioneer ID.
‫Is missing column.
‫And this is just as simple as adding the F test go.
‫We'll just copy this here.
‫So we'll create a new column and we're going to set it to just false because it had no missing value.
‫So df dot head and now.
‫Oops dfe test head wrong data frame.
‫So df test head 102 columns.
‫Beautiful.
‫So if we come right to the end, it should have.
‫AUCTIONEER ID is missing.
‫Wonderful.
‫So this means finally.
‫Now our test dataframe has the same features as our training dataframe.
‫We can make predictions.
‫Let's do it.
‫So make predictions on the test data.
‫Wonderful.
‫So we'll go test spreads equals df test on an ideal model.
‫So our ideal model got predict df test.
‫Boom.
‫That worked.
‫And let's have a look.
‫These should all be like prices, an array of prices.
‫And it's going to be how big is our test data frame?
‫So these are all the sale prices that we've predicted.
‫So 12,457 samples, but it's not going to help us in an array format because Kaggle expects it to be.
‫In a dataframe with two columns sales ID, sales price.
‫So what we can do, we can turn it into that.
‫So format predictions into the same format Kaggle is after.
‫We've made some predictions, but they're not yet in the same format Kaggle is asking for.
‫Now, we'll just link that here evaluation so we know we could go there.
‫Okay.
‫So how would we do this?
‫Well, we need the sales ID column and we need the sales price prediction.
‫So what we might do is just make a simple dataframe df pred equals pd dataframe.
‫So empty dataframe df spreads the sales ID column is just going to be sales ID from DF test so we can
‫go equals DF test sales ID.
‫Oh, that needs to be the same capitalisation.
‫And then the sales price is going to be test spreads.
‫So just our array here and then let's have a look at DF spreads.
‫Boom.
‫There we go.
‫Look at that.
‫How exciting.
‫We've got a submission that we could submit to this competition.
‫But at the moment, this well, this cable competition is no longer running.
‫But that's an example of how you could get your data in a format that it's asking for and then to export
‫that.
‫How might you do that?
‫Well, if we wanted to export this to CSV, we might do something like DF Preds dot to put a little
‫comment here.
‫Export prediction data to CSV.
‫We'll put it into data and we'll go test predictions.
‫Something easy dot csv and index equals false.
‫Wonderful.
‫Or actually, no.
‫This is Blue Book for bulldozers.
‫We'll see if that'll export.
‫Let's go up into our data folder.
‫Do we have that test predictions?
‫CSV.
‫Wonderful.
‫Okay.
‫And so now we've got some test predictions.
‫We can't really evaluate these though, because we don't have the ground truth labels for the test data
‫set.
‫That's why we were doing our evaluation on the validation data set.
‫So what we might look at one final thing to wrap up this project is feature importance.
‫So we've just used our the patterns.
‫Where is it here?
‫When we made predictions on the test data, we've used the patterns that our model has found in the
‫training data set to make some predictions on sale price.
‫And these patterns are how each of these columns contribute to predicting the sale price.
‫So a logical thing or a good idea to figure out would be in feature importance which what columns here
‫or what features meant the most when the model was trying to make a prediction.
‫So let's do that in the next video.