‫Okay, great.
‫Now we have a little helper function that not only returns us our images in the form of tenses, we
‫get processes, the images for us, but it also returns the label so we can get it back in a tuple form.
‫So let's utilize that to write another function that's going to turn our data, all of our data, because
‫this only does one sample.
‫We need a way to turn all of our data so our X and Y, our data and labels into batches of size.
‫32.
‫Alright now we've got let's write that down.
‫Remember we're trying to communicate to our selves here in notebooks a way to turn our data into tuples
‫of tenses.
‫And we go here in the form.
‫Image label.
‫Let's make a function to turn all of our data.
‫X and Y into batches.
‫All right.
‫Wonderful.
‫Now, again, if you're wondering where all of this came from, a great place for any kind of data loading
‫is to read through the documentation.
‫I know it can be a bit boring sometimes, but trust me, reading through even if you don't understand
‫the first time.
‫I didn't understand the first time.
‫Read through it.
‫Try it out.
‫Try it out again and see how you go.
‫This is going to be.
‫A big dog of a function.
‫Oh, that's fitting because we're working on dog vision, but it's going to save us a hell of a lot
‫of time later on.
‫So let's do it.
‫So we need to define the batch size.
‫So 32 is a good start.
‫Remember TensorFlow defaults to 32 a lot.
‫So if you don't set it somewhere, it'll probably default to 32.
‫But we're going to define it anyway.
‫So let's create a function to turn data into batches, and now we're going to write through this function
‫together.
‫And it's going to do a few little things differently depending on if we want a training data batch,
‫a validation data batch or a test data batch.
‫Because remember, these are our three sets there.
‫So these are our three sets.
‫So we've got a training set.
‫We're going to turn that into a training batch, a validation batch and a test batch.
‫All right.
‫So we want def create data batches.
‫We want it to take in an X and a Y.
‫And actually why can be none.
‫Now I want you to think about why.
‫Why might be none.
‫Because why is our labels?
‫This will probably make sense in a little bit.
‫But just have a think about why might we want to create a data batch with no labels?
‫Valid data equals false and test data equals false.
‫And we'll leave ourselves a little docstring here.
‫I used to have a colleague called Ron and he told me a technique called rubber ducking.
‫He was a lot better coder than I was and he said, If you're having trouble.
‫Figuring out how to do something in code, write it out in language first, or say it to yourself in
‫language as if you're talking to a rubber duck.
‫And rubber ducks don't understand English very well, so you have to make it very clear.
‫That's a great tip from Ron.
‫Thank you, Ron.
‫So this function creates batches of data out of image X and label y pairs.
‫It shuffles the data if it's training data, so mixes the data around because if there were some inherent
‫order into the training file that we downloaded, probably not because these look pretty jumbled up.
‫We don't want our model to remember that order.
‫We want it to be as adaptable as possible no matter what order and image comes in.
‫But so if it's training data shuffle but doesn't shuffle if it's validation data.
‫Now I want you to have a think about why it might not shuffle if it's validation data.
‫Don't worry, we'll cover that, but just have a think about it for the meantime.
‫Also accepts test data as input.
‫Now here is why.
‫Why might be none.
‫Because test data doesn't have labels, of course.
‫Now.
‫What we want to do is build a function that kind of does the same thing but has some if clauses or elif
‫clauses, if it's valid data or test data.
‫So we know what we kind of want to do.
‫And we've got some conditions here that if it's valid or test data.
‫So let's start off with a test data because I think that's the least amount of things.
‫So if the data is a test data set.
‫We probably don't.
‫I was about to write, definitely had labels where we know we don't have labels if it's a test data
‫set.
‫So we'll go if test data.
‫Print.
‫Let's give ourselves a little, little heads up.
‫I love giving myself a little heads up.
‫What's going on?
‫Creating test data batch is just to make sure that later on, hey, maybe we're making some data batches
‫and we've accidentally passed a false parameter to test data and it ends up doing it differently for
‫the test data.
‫We don't want that.
‫We want to know that it's making test batches.
‫So data equals TFTP data data set.
‫So this is the module in TensorFlow data.
‫We're making essentially a batch data set and we're going to this is a little interesting one here from
‫tensor slices.
‫So it creates a dataset, a.k.a. just a TensorFlow dataset whose elements are slices of the given tenses.
‫So basically what this says is pass me some tenses and I'll create a dataset out of that.
‫Little confusing, I know, but this is just how it's done in TensorFlow tf constant x.
‫So basically what this says is I'm going to create a TensorFlow dataset.
‫From the tensest slices.
‫X and because we're passing X to T constant, it turns X into a tensor.
‫So this is only file paths.
‫No labels.
‫Okay.
‫And now data back.
‫Here's where we turn our TensorFlow data set, because right now this would turn all of X into a data
‫set.
‫We need to turn it into a batch size of 32.
‫So equals data format.
‫We need to process the image.
‫And we need to turn it into a batch batch size.
‫Whoa.
‫A lot of stuff going on here.
‫Daniel, what's happening?
‫Well, we've talked through the line above, but this one, all it's saying is basically take this data,
‫which is a TensorFlow data set made of X.
‫Which is just file names in the form of tenses and then map our process image function.
‫See, this is why we made it into a function so we can call it process image.
‫It's going to go through all of these steps here on our file names.
‫So it's going to import them, turn them into normalized tenses, and then turn it into a batch of batch
‫size.
‫Let's go here.
‫So TensorFlow data batch.
‫This is the workflow I use whenever I'm looking at.
‫A function.
‫So if we go back.
‫There we go.
‫Batch.
‫Combines consecutive elements of this data set into batches.
‫There we go.
‫So if we pass it a range of eight and we want it to be a batch of three, it's going to split eight
‫into three batches.
‫So that's just exactly why is this in a new tab?
‫That's what it's going to do here, except as of size 32.
‫Wonderful.
‫So that's the test data.
‫Now we want to do if the data is a valid data set, we don't need to shuffle it.
‫Now, I want you to think about again, why would we not need to shuffle it?
‫It's okay if you're not entirely sure.
‫I got caught out on this a lot of times, too.
‫So we'll print a little progress statement here and then we'll do much the same as this.
‫We could just copy that, but we're going to practice writing it out.
‫TFF Data.
‫Data set from.
‫Tensor slices.
‫Now, this is where.
‫Because our validation set has labels.
‫We're going to pass it a couple of X and TF constant y.
‫So just turning our file names and labels into tenses.
‫And we got here file paths.
‫Labels.
‫So now we have a TensorFlow data set.
‫Of image file pots and labels because it's a validation data set.
‫The validation data set has labels and we're going to turn it into a data batch equals data map, just
‫the exact same as above.
‫Process the image and turn it into a batch of batch size.
‫Oh.
‫Getting trigger happy.
‫Wonderful.
‫And then return data batch.
‫And finally, we want if none of these work well, if valid data is false and test data is also false,
‫it's obviously going to be a training batch.
‫So we're going to go create training data batches.
‫Bom, bom, bom.
‫And let's go.
‫So if it is a training data set, we want to shuffle it.
‫That's the only difference here.
‫So we want to go turn file paths and labels into tenses.
‫So just as we done with the validation, we could just copy this.
‫Now, there's probably a more efficient way to write this function that does the same functionality,
‫but I've tried to write it in a way that is fairly understandable as to what's going on.
‫Constant X and as I said that I might have said that it's fairly understandable.
‫But if you're coming across something like this for the first time, trust me, when I did, I was confused.
‫But once I went through it step by step, remember our technique for breaking down a function?
‫Write it out line by line, and then check out what it's doing at each line.
‫So we want to shuffle the path names and labels before mapping.
‫We'll go into this in a second before mapping image processor function is faster than shuffling images.
‫I'll just write the code so we can talk about it.
‫But data shuffle.
‫Buffer sighs.
‫So buffer size just stands for how many variables do we want to shuffle them?
‫We want to shuffle the whole lot.
‫Well, a lot going on here.
‫Well, all this is saying is, hey, take this data, which is a TensorFlow data set of the data, so
‫the file names and the labels and then shuffle it up and we want to shuffle the number of examples.
‫If we only shuffled 100, it would take the first 100 examples of X and Y and shuffle them.
‫But no, we want to shuffle them all.
‫And the reason being, you might see other tutorials shuffling later after we run the map function it
‫goes.
‫Data map.
‫Process.
‫Image.
‫So if you did shuffle after you've already processed the image, it takes a lot longer to shuffle a
‫full image than it does just to file name.
‫So as I said, remember, we're trying to minimize the time between experiments.
‫So we want to try and get any speed ups that we can.
‫I've just realized we've made a little mistake up here.
‫In our validation data set because we have.
‫Images and labels.
‫We need to map the function, get image label, which also processes our image.
‫Whereas for our test data set, because we have no labels, we can run directly the processed image
‫function.
‫So let's let's replace this get image label and it's going to be the same.
‫For our.
‫Training dataset, so get image label wonderful.
‫And it might just write out what this is doing with a little, little comment.
‫So create image label tuples.
‫This also turns the image path into a pre processed image.
‫So if you can.
‫Shuffle your data when it's in its smallest format, a.k.a in our case file names rather than full images.
‫And now finally, we want to turn the training data into batches.
‫So data batch equals data batch.
‫Bash sighs.
‫And then we'll go out here.
‫Return.
‫Data bash.
‫Wolf, that's a bit of a behemoth of a function.
‫But as I said, this is going to save us a lot of time later on and you'll see that.
‫Remember, this is the kind of development work style I want you to start thinking about.
‫Hey, am I going to reuse some functionality later on?
‫Well, if I am, I'm going to write it into a function.
‫And you probably already know that because you're a lot smarter developer than what I am.
‫I used to just fluster around and just write things line by line.
‫Now let's test our function by creating training and validation data batches.
‫So we'll go train data.
‫We're going to use our big dog function just above.
‫We're going to pass it X train for the training data and Y train for the validation data and that's
‫all we need.
‫And then for the VO data, we can go create data batches.
‫X Val.
‫Y Val is the is it valid data?
‫Equals true.
‫And now, fingers crossed.
‫We should see.
‫Our print progress messages.
‫Yes.
‫That is amazing.
‫Okay.
‫Now, let's check out the different attributes of our data matches.
‫So if we go try data, it's now in the format of a data batch which is TensorFlow is preferred way of
‫processing things.
‫There we go.
‫This is our training data, the first one here.
‫What's it saying?
‫So you've got shape.
‫None.
‫Two, two, four, two, two, four, three.
‫This is our image.
‫None is for batch size.
‫I know we've said it to 32, but it's going to stay as none because batch size, remember, is flexible.
‫So this is just our images are now in batches of 32 and they've got a shape of two, two, four, two,
‫two, four, three.
‫Height, width, color channels of type two, float 32.
‫And we've got a tuple here of our label.
‫So these are image label pairs in the form of tenses, and our labels also have a batch shape of none
‫because batch size is flexible.
‫And they have a dimension of 120, which is because if we go, why zero?
‫There's 120 different dog breeds, and then it's the same again for the validation data.
‫Beautiful.
‫So that was a pretty in-depth one.
‫Once his video is over, which will be in a few seconds, I'd go back through and just read through
‫what's going on in this function here.
‫If you're not entirely sure, I'd look up loading an pre-processing data tutorial of images, have a
‫read through that.
‫And then the best way to learn is to really just write this out.
‫Break down the function and try to run line by line.
‫That's how I break down things for myself.
‫But the way our data is at the moment, it's still kind of hard to understand if you're new to the concept
‫of batches.
‫It's probably a very difficult concept to grasp.
‫It's perfectly okay not to know what's going on right now.
‫So to help ourselves understand, rather than just having this this gobbledygook being printed out when
‫we're trying to check out what's in our data batches, let's write a function which is going to help
‫us visualize what's going on.
‫So that's what we'll do in the next video.