‫Who?
‫We have been through a lot.
‫But we're going to persevere.
‫We're going to get through this together.
‫We've got a few more functions to go through.
‫So the next one is sample.
‫Now, this is a great way to to shuffle your data frame.
‫What I mean by that is mix up the indexes.
‫So in future videos, when we start applying machine learning algorithms to to our data.
‫One of the most important steps is creating a training, validation and test set.
‫And one of the steps to doing that is randomizing the order.
‫Your data comes in because what you want to avoid is the order that these samples come in may have some
‫kind of influence on patterns in the data.
‫When our machine learning algorithm is learning things, we want it to be as general as possible.
‫So we don't want it to care about the order.
‫Now there are some exceptions to the rule with time, series data and whatnot, but I'm just going to
‫show you this example.
‫If you wanted to shuffle rows in a data frame, so let's do car sales dot sample.
‫And now sample is basically saying, take a sample from this data frame with this parameter, frack
‫is the fraction of that sample.
‫So it's going to be in an integer form.
‫So for example, 0.5 is 50% of the data.
‫So it's going to get five rows out of ten.
‫But we in this case want to shuffle everything.
‫So we will use one for 100% of the data.
‫There we go.
‫So now you can see that the index has been shuffled.
‫So this is essentially the same data frame just with the order mixed around.
‫Now, the important thing to remember with PANDAS is that the information in a row will stay with its
‫row.
‫So row nine, Nissan, Nissan White, 31,600 kilometers, four doors.
‫So the rows stay the same.
‫All that's changing with this function sample is the order that things appear in.
‫And now let's have a look at the dataframe again.
‫Oh, silly me we keep forgetting to reassign So we want to reassign here equals There we go Now it's
‫going to definitely shuffle the car sales dataframe or maybe we create a new one shuffled because we
‫want to keep the original one in order.
‫So we've got car sales shuffle.
‫This will return a copy, but car sales shuffled will be in a different order to the original car sales
‫dataframe.
‫Beautiful.
‫Now if we wanted to manipulate this shuffle DataFrame, we could call the same functions we've called
‫before on the shuffled version.
‫And another handy thing about the sample function is that, say, for example, you had like a dataframe
‫with 2 million rows, as is really small, it's only ten.
‫But in practice you're going to be working on data sets with a lot more rows.
‫Sometimes running functions in pandas takes a long time on millions of different rows.
‫What you might want to do is practice on only 20% of the data, only select 20% of data.
‫And now this number could be arbitrary, right?
‫In our case, we've chosen 20%.
‫So that's giving us two different rows.
‫If you had 2 million rows, maybe you want to practice on 1% of the data.
‫So that's still 20,000 rows.
‫And so that will allow you to do lots of different experiments a lot quicker than doing it all on 2
‫million rows at one time.
‫So that's something you'll have to consider in the future in your projects.
‫When you're working on larger amounts of data, is your computer powerful enough to continually run
‫functions on millions of rows, or do you want to flesh out some little experiments on smaller amounts
‫of data first and then upgrade those experiments as you figure out what works and what doesn't?
‫Now we look at our car sales, shuffled data again.
‫We've got our indexes, but they're all out of order.
‫How would we get those back into order so we can go?
‫Car sales shuffled reset index.
‫This function, as you could imagine, is going to reset the index here.
‫So we want.
‫Boom.
‫And now there's a way here that you can remove this column.
‫So maybe we go in place.
‫Oh, let's have a look.
‫Did that do that in place?
‫Shuffled.
‫Because by default, the reset index column adds the old index as a new column on the very left.
‫And so we've got order numbers here, but this is the shuffled index that it's made a column.
‫Maybe we don't want that.
‫I think we can get rid of that by in place equals true.
‫Remember a lot of panda's functions have the in place parameter.
‫Shuffle in place equals true.
‫Oh, it still has that.
‫Hmm.
‫Well, the way we figure this out is if we look up the documentation for reset index, let's do that.
‫Pandas reset index.
‫I don't want that extra index column there.
‫So let's see.
‫Drop.
‫Do not try to insert index into dataframe columns.
‫I think that's what we need.
‫Is there an example drop equals true?
‫Yeah.
‫I think that's what we need.
‫So we want in place equals true.
‫And we want drop.
‫Equals true.
‫Drop is false by default.
‫Yeah.
‫Default false.
‫So this is another example of how you can look out what a function does in pandas.
‫I just googled the function name reset index.
‫Again.
‫You won't know these things from the start, but it's not about knowing things off by heart.
‫I don't know everything off by heart.
‫It's about researching different ways that functions work and trying things out.
‫If in doubt, run the code.
‫Let's do that.
‫We still have that there.
‫Maybe we come back up here.
‫We're going to reset this.
‫Reset this.
‫There we go.
‫We had to reset our car sales, shuffle dataframe and now we've set drop to equal true.
‫We don't have that index column on the left here.
‫There's one more thing we're going to look at, and that's how to apply a function to a column.
‫So let's say we have one.
‫We'll look at our car sales data frame.
‫Remember, this is the process you're going to be taking a lot is viewing your data frame and then manipulating
‫it, viewing it, manipulating it, viewing it, manipulating it.
‫So it's looking pretty prime at the moment, but we want to convert our odometer to miles.
‫So how would we do that?
‫Car sales first we have to select it.
‫You can do that by typing in the column name and then we're going to reassign it.
‫We remember to reassign it this time.
‫Odometer kilometers.
‫Yep.
‫Now this is the apply function.
‫Apply lets you apply some kind of function, whether it be numpy or lambda to a certain column.
‫So let's just type this in.
‫And lambda functions can be confusing to begin with.
‫Took me a while to learn them, but we're going to step through this one and figure out what it's actually
‫doing.
‫So if and out run the code, we want carsales to view the whole dataframe.
‫Run the code.
‫There we go.
‫Now.
‫What do you think this is?
‫Just these numbers have changed.
‫We've got 150,000.
‫It is now 93,776.
‫Well, let's step through this odometer column.
‫Tick.
‫I don't want to call them tick.
‫Apply.
‫Yep.
‫Lambda now, lambda is a keyword in Python, which is short for an anonymous function.
‫So basically this is saying apply this function to X divided by 1.6.
‫So that's what this is saying.
‫Apply.
‫Reset X to be equal to x divided by 1.6 y 1.6 because conversion from kilometers to miles.
‫He goes about 1.6109, but I've just rounded it to 1.6.
‫That's going to divide our number of kilometers by 1.6 because in this case, X is the values in our
‫odometer column.
‫So this little line here is going to look at this column and go every time it's going to pick up this
‫150,000, which is X and then divide it by 1.6 and then reassign it to that value.
‫So the first column, 94,000, thereabouts, first row, sorry, second row, 87,899 is actually about
‫55,000 miles.
‫So that's what this apply function is doing.
‫And now this is just a simple example of a little lambda function, but you can get very elaborate with
‫the things you put in apply.
‫We're not going to dive too deep in that for now.
‫Just want you to know that apply is a way that you can assign a function to a column.
‫So you might have written another function somewhere else and then use apply to apply it to a column.
‫Whew.
‫We have been through an absolute mountain of information here.
‫Don't fret.
‫You don't have to know all of this off my heart to begin with.
‫Remember the stamps.
‫Try it.
‫Run your code.
‫If in doubt, search for it.
‫Try again.
‫And then if you're still stuck.
‫Ask.
‫In the meantime, before we get into the next section, this is going to wrap up the PANDAS section.
‫Check out the resources section for a little bit more documentation about PANDAS.
‫Some exercises you can try out for yourself.
‫But otherwise, if you're with me, let's jump in and learn a bit more about some other tools that we
‫can use for data science and machine learning.