‫By now you're probably thinking, my goodness, there is so much to remember and I'm not going to lie.
‫I felt the same way when I first started.
‫I went through some information on pandas and realized, Holy goodness, there is well, I didn't actually
‫say goodness.
‫I said a different word there.
‫You could probably imagine what word.
‫When I said, there's so much here, how am I going to remember all of these functions of my heart?
‫And the point here is to remember you don't necessarily need to know all of this off by heart.
‫The only reason I can kind of do this on the fly is because I've had a little bit of practice.
‫Don't forget to always try the code yourself.
‫You can look it up, find things like this.
‫This is part of the the research part of being a data scientist, part of being a machine learning engineer
‫is learning how to ask different questions.
‫Think about how can I do this with the data?
‫And then trying to find out.
‫With that being said, let's have a look in this next little section here at how to manipulate data
‫with pandas.
‫So we'll put another little heading like we've been doing, maybe manipulating data that sounds a bit
‫better this game and press m for markdown because we're staying communicative.
‫Excellent.
‫And now again, the beautiful thing about pandas is that if you can think of it, you can probably do
‫it.
‫So the first thing we'll see is manipulating data is some string methods.
‫So Python has a bunch of different string methods like lower.
‫What do you think this will do if we go castles make dot string dot lower?
‫Let's see.
‫Shift and enter.
‫Well, it's lowered the string because its data type is an object, according to pandas, but it's actually
‫also a python string.
‫So anything that you can do on strings in Python, you can do on string columns in pandas.
‫There's a bunch of different functions there, but we'll just remember that you can access the string
‫value of a column by doing dot string.
‫We did that above in the price column.
‫We converted it from a string to an integer.
‫Now if we have a look at our car sales data frame again.
‫What they.
‫We didn't.
‫We just lowered the string.
‫But it didn't actually save.
‫Now, here's an important concept to remember.
‫It's that Pandas requires if you want to change a column, it requires reassigning that column.
‫So that means we want to set car sales.
‫Make equals car sales make we want to do not shrink, not lower.
‫There we go.
‫Car sales make?
‫Yep.
‫Don't shrink lower.
‫Now, what this will do is it will reassign the car sales, make column to be a load string.
‫So let's have a look.
‫There we go.
‫The string has been lowered.
‫So now all the letters in here are lowercase.
‫You'll probably come across this a fair few times when you're working on different data frames.
‫Is this this concept of reassignment, what some functions have is that to save you from having to go
‫this column equals this column.
‫Again, some sort of function is some functions have a parameter called in place.
‫We'll have a look at that in place parameter along with some missing data.
‫But you might be saying, hey, our car sales data frame doesn't have any missing data, but in practice,
‫in the real world and when you get real data sets, I've made this one up, right?
‫So I've typed in this into here.
‫Real data sets often contain missing data, so they often look like this because you might have someone
‫tracking some records.
‫Ideally they'd look like this, all the data there.
‫But in reality, when you get a new data set, sometimes it's got a few holes in it, right?
‫Someone hasn't tracked everything correctly.
‫And that's that's just life.
‫You don't always get perfection.
‫So we're going to.
‫Do you remember how to import CSV file.
‫I've already exported this one so we're getting paid a dot read csv and we're going to do tab auto complete
‫car sales, missing data csv.
‫What this is going to do is save the car sales, missing CSV and import it as a pandas data frame.
‫So all it's going to do is bring this into being a data frame.
‫Shifting into.
‫Now, this is how PANDAS describes missing data and a n for nan.
‫That's basically just saying, hey, I've got no value here.
‫Not a number or something like that.
‫Actually, don't.
‫Not entirely sure what Nan stands for, but all I know is that it just means there's no value here.
‫All right, so what if you wanted to fill these missing values with something?
‫So, say, for example, we wanted to fill the odometer, so we're accessing that.
‫There's a function called fill a which is fill these nan values and you can pass fill in a some kind
‫of value and it will fill these values.
‫So maybe we want to fill up the odometer column with the mean of the rest of the odometer, which in
‫reality isn't a very good practice.
‫But just for demonstration sake, maybe let's let's do that.
‫Don't mean.
‫So what this is saying.
‫Well, no, we want missing because this is our missing, missing, missing.
‫There we go.
‫What this is saying is grab the Carsales missing odometer column this one, and fill in a fill the missing
‫values with the car sales, missing odometer columns, main value.
‫So the values that do actually exist there, we're going to get the mean and fill it.
‫Let's quickly see what that is.
‫Car sales look missing odometer.
‫Don't mean.
‫So if we do this correctly, should fill them up with this value.
‫Let's go there.
‫Beautiful.
‫Now we've got all those missing values.
‫The third one down is now filled with the car sales.
‫Missing odometer mean value.
‫We got a few rare occurrences there of all the missing ones.
‫So now if we have a look at the data frame and this is a practice you'll be going through, right?
‫You'll be making manipulations here and then you'll be consistently viewing the data frame.
‫So you might do head, but because it's only small, we'll look at the whole thing.
‫Oh.
‫What's happened here.
‫There's still Nam.
‫This is a problem.
‫This is what I was mentioning before.
‫That in place value that we can use.
‫So let's see it in action.
‫So we could do a reassignment.
‫By going car sales.
‫Missing odometer.
‫Equals that.
‫And then re assigning the values to being something like that.
‫But what we'll do will look at the in-place parameter.
‫So you can see what I mean.
‫So we'll get rid of that.
‫We're going to copy this line of code so we don't have to retype it out.
‫But remember, you should always practice retyping code so you get used to it.
‫Now this fill in a has a parameter and lots of pandas functions have this in place equals it's false
‫by default, but we want true.
‫Actually, this is a good time to look at the documentation.
‫So.
‫So we didn't know this.
‫Fill in a function.
‫What would you do to figure that out?
‫So maybe we go to Google and we go Pandas.
‫How to fill missing data in a column.
‫Working with missing data.
‫Yes, this is a PANDAS documentation you get very familiar with.
‫Looking at these when you first start looking at this documentation can be a bit overwhelming because
‫there's a lot going on.
‫Remember, this documentation is for every single thing that pandas can do.
‫So going through it can take a while.
‫Well, if we're looking here, maybe we want to search for Phil.
‫Fill fill missing values.
‫There we go.
‫Phil, in a now often it comes with let me zoom in for you a little hyperlink to the exact function.
‫Phil and I can fill in any values with non NA data in a couple of ways.
‫Excellent.
‫And then give some examples here.
‫We're going to click on the fill in a function and see what happens.
‫So we got DataFrame fill in a, which is what we've got so far.
‫DataFrame or Domino because we only want the odometer column, not fill in a yellow tick for that value
‫equals none.
‫So by default the value equals none.
‫But it can take a few different values.
‫In our case, our value is the main of the odometer column.
‫We'll come back.
‫We can see this little in place parameter.
‫It's got a few others, but we're not going to cover them for now.
‫So in place bull by default.
‫False.
‫If true, fill in place.
‫Note this will modify any other views on this object, e.g. a no copy slice for a column in a dataframe.
‫So if we set in place to true, it means fill in place.
‫Let's have a look.
‫In place equals true.
‫Before we do, we'll take one last look at what our data frame looks like.
‫Still missing values.
‫So let's see what happens.
‫Phil and I in place equals true.
‫Now.
‫Let's have another look.
‫Go missing.
‫Ah, there we go.
‫So we've filled the values without reassigning.
‫So without typing car sales, missing odometer.
‫I don't want to.
‫He calls that.
‫So that's just a thing to remember.
‫You can always do it this way.
‫Remember how I said Pandas has multiple different ways of doing the same thing?
‫One way to reassign values is to reassign them by putting the exact value on a replace at the start,
‫or by setting the in-place parameter of a function to true.
‫So that will really depend on how your workflow goes.
‫Personally, I'm not a fan of it in place.
‫I like just leaving it by default and then reassigning the value at the start.
‫Let's have a look at one more thing before we will take a little break as this lecture is getting a
‫bit long.
‫It's if we wanted to remove these missing values, so say, for example, we're happy with filling the
‫odometer column.
‫But every other row that has missing values, we just want to ignore them because missing values aren't
‫going to help us with our data analysis.
‫In truth, they might.
‫But we want to just see how we would get rid of them.
‫Carsales missing.
‫Now there's a function here called drop and a now we might just see what happens if we just run that.
‫Drop in a beautiful now if we type in here car sales missing, what does our dataframe look like?
‫Oh, it hasn't happened again.
‫Remember, we've just covered this.
‫What do you think we can do to fix this up?
‫So we might do car sales drop in a in place ankles true.
‫Oh, hey.
‫Shift into and we'll have a look at our data frame.
‫Car sales missing.
‫Beautiful.
‫Now, what do you think the value is in maybe not putting in place?
‫That's true.
‫Now our car sales missing data frame is always going to return.
‫Those rows with missing values are going to be dropped.
‫What if we wanted to re access them?
‫We'd have to re-import the C as V.
‫So maybe we do that.
‫Equals PD raid sas v car sales missing.
‫So we'll reset our data frame.
‫We'll have a look at it.
‫This one's going to have missing values.
‫So maybe we create a new dataframe carsales missing dropped equals carsales missing dot drop n a in
‫place equals true.
‫So what this will do is it returns a copy of our car sales missing data frame, but with dropped values.
‫Oh, I said in place equals true.
‫We have to rerun this.
‫No, that's okay.
‫So sometimes you make mistakes here.
‫We're going to go drop and there we go.
‫Car sales missing, dropped.
‫Beautiful.
‫So it's going to be the same data frame, but dropped the missing values.
‫And then what we might do is car sales missing.
‫Dropped.
‫Save this to to CSV as.
‫Carsales missing dropped csv.
‫That way we can.
‫If we did want to access the rows with missing values we can from our car sales missing data frame this
‫one here.
‫Or if we wanted to manipulate our data frame with missing values dropped, we can from this data frame.
‫So that's the important distinction there between using in place or not using in place.
‫Alright.
‫We've covered enough on manipulating data in this lesson for the time being.
‫So take a little break.
‫Revise over the few functions that we've gone through, have a trial out with in place versus not in
‫place.
‫And I'll see you in the next video.