‫We've got a predictions array, but now we need to get it in the Kaggle submission format.
‫So that's what we'll be working towards in this video.
‫Let's go make ourselves a little heading preparing test data.
‫Set predictions for Kaggle.
‫And if we look at the sample submission, we kind of know what we need and we know we need an ID column,
‫which is the file IDs from our test images.
‫We know we need columns for each of the different dog breeds, and we know we need the prediction probabilities,
‫which is what we've got at the moment.
‫We've got an array with all of these numbers.
‫We basically need to just set up our predictions probability array into a dataframe like this, export
‫it as a CSV and then upload that to Kaggle.
‫So let's write that down.
‫Actually, we'll put a little link in here.
‫Of what Cargill wants.
‫Evaluation.
‫Copy that.
‫This is just what the sample submission has to look like.
‫Looking at the Kaggle sample submission, we find that.
‫It wants our models.
‫Prediction probability outputs in.
‫A DataFrame with an ID column and a column for each different dog breed.
‫Well, it's not really a dataframe, it's a CSV, but we can be a little bit incorrect there in our
‫wording.
‫And how are we going to do this?
‫Let's write ourselves some guides.
‫So to get the data in this format.
‫Well, this is a testament to our favorite little graphic here of preparing inputs and outputs.
‫So now we've worked with the inputs.
‫We've saved and reloaded our train model.
‫Now we're making a bunch of predictions that we have made predictions.
‫We're preparing our outputs, most specifically preparing our outputs for Kaggle.
‫We will create.
‫A pandas dataframe with an ID ID column, as well as a column for each dog breed.
‫And then we're going to add data to the ID column by extracting the test image IDs from their file paths.
‫And then we're going to add data, the prediction probabilities to each of the dog breed columns.
‫And then we want to export the data frame as.
‫A CSV to submit it to Kaggle.
‫This is what we're after.
‫And probabilities is spelt wrong.
‫Standard Daniel.
‫Typo.
‫KING Some nice little fall stops.
‫All right.
‫So the first step, let's go through and create a pandas dataframe.
‫We've had experience of this way back in the pandas section with empty columns.
‫So we'll go spreads DF equals PD dataframe.
‫So just a standard nomenclature for a prediction dataframe spreads DF Preds dataframe or predictions
‫dataframe columns is going to be equal to ID.
‫Plus.
‫We need a column for each different dog breed.
‫So we'll do one list unique breeds.
‫And I just want to show you what list unique breeds looks like.
‫List unique breeds.
‫So there we go.
‫So if we added ID to that.
‫Let's give that a about.
‫I'd say, oops.
‫Trigger happy once again.
‫Plus.
‫There we go.
‫ID plus list unique breeds.
‫That is some column names for us.
‫We'll comment out that and we will check out the head of our Preds DF should be empty with just a bunch
‫of column names.
‫That's what we're off to ID and all of our unique breeds in order.
‫We've done this.
‫So putting in some emojis.
‫All right.
‫Add data to the ID column by extracting the test image IDs from their file paths.
‫I believe we have already done that.
‫But what we might do is just do it again.
‫Test file names.
‫But the way we can get ID is by.
‫We want to extract this out.
‫So we could do a little bit of fancy regex on this.
‫But I'll show you another way.
‫Append test image IDs to predictions dataframe.
‫So we want to go test path.
‫Do we already create test path?
‫We did.
‫Okay, wonderful.
‫So let's get test IDs.
‫So this is just the path to our test file and test IDs is going to be equal to.
‫A list comprehension of O's dot path.
‫And then what is it?
‫Split text.
‫I'll show you what this does in a second.
‫Split text and for path in o's dot list dir test path.
‫That should be correct.
‫How can I show you an example?
‫Okay.
‫Stop path, split text.
‫We want to get test file names.
‫Zero.
‫Well, this work.
‫There we go.
‫Okay.
‫So it splits off the last little bit of the path, but we only want this.
‫So that's why we have to do it for just the test path.
‫So we want to just go through this test folder.
‫Now, it's not going to show up here, is it?
‫Because there's a lot of images in there that will take a long time to load.
‫And CoLab, you're just going to have to trust me with what this does.
‫Let's just see it in action anyway.
‫Test IDs.
‫Will this work?
‫If in doubt, run the code.
‫All righty.
‫Actually we don't need the JPEG bit.
‫So what we're going to have to do, we only want the first element of each of these.
‫So we're going to have to put a little zero index there.
‫And let's see what this does.
‫That's what we're after.
‫Test IDs.
‫I don't want a large print out there.
‫So what I'm going to do is just go press.
‫DF So I'm taking my data frame from up here.
‫I'm going to make an ID column with the test IDs.
‫So just taking that and putting that there, a little bit of overkill running this cell three times.
‫But that is all right.
‫Let's check out the head of our DF, as we always do when we manipulate our data.
‫Okay.
‫So we've got some ideas in there.
‫Now we need to just append all of our prediction probabilities for each dog breed.
‫Now, how might we do that?
‫Hmm?
‫I want you to have a think about this for a second.
‫What do we have?
‫We have test predictions, which is an array of predictions, probabilities and.
‫What do we have?
‫That's a list of different dog breeds.
‫We have unique braids.
‫Of course we do.
‫So add the prediction probabilities.
‫This is getting exciting.
‫We're getting so close to being able to submit our first full blown, deep learning model to Kaggle.
‫We've gone from head to toe, the whole process, the whole shebang, so you can't tell I'm pumped.
‫So this is going to assess the breadth df of all of the unique breeds, so all of the breeds columns.
‫And then we want to just set that.
‫Look how easy this is to test predictions spreads df dot head.
‫And we are done.
‫Ladies and gentlemen, have a go at that.
‫We would compare that to the sample submission.
‫Of course, this goes on for a lot longer.
‫But ours is using the model we have built.
‫The numbers are going to be different, but that's okay.
‫Now let's export it to a CSV so we can submit it to Kaggle.
‫How do we export pandas dataframe to csv?
‫We use to CSV.
‫Of course we want to go drive my drive.
‫I'm going to go to dog vision and I'm just going to save it as maybe fall model predictions submission
‫one CC or maybe we'll tell ourselves my battle.net.
‫Way to.
‫Dot csv and we don't want index.
‫So index equals false.
‫We'll go here.
‫We'll just write what we're doing.
‫Save our predictions dataframe to see us v for submission to Kaggle.
‫Is this going to save?
‫Fingers crossed.
‫Yes, of course it will.
‫Now, let's have a look in here.
‫In our files.
‫Oh, maybe we've glitched out.
‫We've just saved a CSV there.
‫Maybe car labs lost the plot on us.
‫Give me a second for my car lab window to reload and I'll be back once I have access to this folder.
‫Okay.
‫I'm back.
‫So sorry about that little interruption in broadcast.
‫It seems my lab window decided to go bonkers iec y because I decided to open the test folder and I need
‫to close that and I might have to pause it again.
‫1/2.
‫All right.
‫Third time's a charm.
‫We are back.
‫So be very careful with folders that you open in your collab window.
‫If they're full of data, a.k.a 10,000 plus images, it will freeze up your colab notebook.
‫There we go.
‫For model predictions.
‫Well, we can't really see the whole thing there.
‫Oh, there we go.
‫And that's just our CSV file there.
‫So if we wanted to, we can download that we do want to.
‫So we'll just wait for that to go through.
‫I'll speed up my video until this is fully downloaded.
‫All righty.
‫So this is fully downloaded.
‫All I did was just click download and then waited for it to go.
‫Took about a minute or so, not too long.
‫This is about a 30 megabyte csv.
‫If we come to my downloads folder.
‫If we come here.
‫Yeah, about a 30 megabytes ESV, so 10,000 predictions.
‫Now we've got that.
‫Let's go to Kaggle and we can go late submission because this competition has already closed.
‫You can see my past won there five days ago.
‫So let's upload.
‫I want to show you that this works.
‫Open.
‫I'm just selecting the CSV we just downloaded.
‫It's going to go through.
‫Briefly describe your submission.
‫Let's go transfer learning.
‫That's what we call it, because that's what we used to transfer learning mobile net.
‫They too?
‫Dog vision, baby.
‫And then we'll make submission and it's going to score it.
‫Now there's something up with this.
‫Oh 20.
‫So that would be our goal is to beat this one on the next one.
‫And if we go back to the leaderboard.
‫See where we appeared.
‫We'll probably find public leaderboard scores zero.
‫Hmm.
‫I'd be very skeptical of any competition where people have score of zero.
‫So what you can actually find out is you can actually jank or actually hack the score here.
‫So don't worry too much about submitting to here on a late submission.
‫Your next goal is just to reduce your own score.
‫And if you want my example submission CSV, I'll just link this in the results section.
‫You'll probably get the same score as me.
‫Our next step.
‫In this project.
‫If we were to keep extending it and trying a new model would be to beat our original score rather than
‫paying attention to the leaderboard.
‫Straight away is focus on when you first begin beat your own score.
‫So now we've seen how to make full blown prediction for the Kaggle competition.
‫But that's not our original focus.
‫Well, it kind of was.
‫I lied right back up the top.
‫What's this notebook called?
‫Dog Vision.
‫So our problem is identifying the breed of a dog given an image of a dog.
‫When I'm sitting at the cafe and I take a photo of a dog, I want to know what breed of dog it is.
‫So.
‫It's likely you want to be able to make predictions on your own custom images.
‫So rather than just images that Kaggle has given us in a test and training set, what if you had some
‫images of your own dog?
‫Which I happen to have or a dog that you saw at the cafe that looked really cute and pretty and you
‫wanted to know what breed it was.
‫So you took a photo and you came back to your machine learning model and this notebook and you're like,
‫I want to see what breed of dog that was using my model.
‫Well, let's have a look at how to do that in the next video.
‫Making predictions on custom images.