‫All right, let's fill some of those missing values, shall we?
‫So we'll make a little heading here for all missing values.
‫Beautiful.
‫And we've got a fair few missing values here.
‫So what we might do, I think we'll do the numeric missing values first.
‫So let's put it here.
‫So another little heading and we'll go, Oh, let's change this to number two, just because I like
‫the aesthetic and we'll go fill numeric missing values first.
‫Wonderful.
‫Now how do we check what columns are numeric and what columns are aren't numeric?
‫So what we might do.
‫Remember the is string date type.
‫We did that before.
‫We looped through our data frame right up here.
‫We could probably press shift into there.
‫Where is it?
‫So is String D-type.
‫So there's another one, as you might have guessed.
‫There's also is numeric data type.
‫Is it up here?
‫No.
‫Might be here if we go.
‫Here is numeric.
‫Here we go.
‫Check whether the provided array or d type is of numeric d times because we want to fill numeric data.
‫We want to find out which columns are numeric.
‫Let's loop through four label content in the template item.
‫So we're going to loop through the column names as well as what each column contains.
‫And we're going to if API dot types.
‫DOT is numeric data.
‫Does that make sense?
‫What we need to put in here, if the content yes.
‫If the content in the column is numeric, we want to print out the labels.
‫So that's going to be the name of the column.
‫Wonderful.
‫So let's check out one of these.
‫We made these ones down here so we know that they're numeric.
‫But let's check out just for fun model I.D..
‫Wonderful.
‫So he's going to be the model I.D. or actually we can check what that is.
‫Model ID identify for your unique machine model EFI model description.
‫Beautiful.
‫Okay, so these are our numeric types.
‫And now what we might do is find out which ones of these have missing values when more focus on at a
‫moment, we want to fill the missing numeric columns with some sort of value check for which numeric
‫columns have null values.
‫Now, how might we do this?
‫Well, first of all, let's start out with four label content.
‫We'll just write exactly what we did before DF temp items and now we want to go.
‫If PDB API types DOT is numeric.
‫I tried to do tab, but it was too many options.
‫Is numeric type content.
‫Wonderful.
‫And then what we might need to do is check there's a feature or a function in pandas called is null.
‫So if we go PD is null.
‫Pandas is no.
‫Here we go.
‫So this detects missing values for an array like object.
‫So we can do this by pandas is null content.
‫That's what we want.
‫So we're doing another if statement here.
‫So PD is null.
‫So this is going to go.
‫Is there null values in the content aka the column in the whole column?
‫Is there null values?
‫But we need to make sure this is sum so total.
‫So if there's any null values.
‫So if this is higher than zero, we want to print the label.
‫This should tell us which columns have no values.
‫All right.
‫Now that we know which columns are which numeric columns have no values, we can.
‫Fill them with something.
‫So what we might do is fill the numeric rows with the median.
‫Hmm.
‫So how about we do that?
‫We can just take this.
‫We'll copy this down here.
‫All right, we're going to go fill numeric rows if the median.
‫Wonderful.
‫But there's one other thing that we have to actually do and go here.
‫Well, you don't have to do it, but we're going to do it anyway.
‫And I'm going to write it out first, because we're going to add a binary column which tells us if the
‫data was missing or not, that comment's a bit long, but that doesn't matter.
‫Why do you think we might do this?
‫I'll just let you have a think about it while we do it and we'll go here is missing equals.
‫PD is now content.
‫Wonderful.
‫And then what we're going to do here is fill missing numeric values with medium DF temp label equals
‫content.
‫They'll fill in a content dot medium.
‫Who?
‫We only built upon what we've done before, but we added two new steps in here before we hit shift and
‫enter.
‫On this, we're going against our rule.
‫If F and outrun the code, I want you to just think why we might put this column here.
‫You might be asking us, well, why are we filling it with the median?
‫Why not the main?
‫Well, the reason why we're adding this column here is because if we just fill up all of our missing
‫data with the median, then there might be a reason for that missing value that was there.
‫We're keeping that information.
‫The fact that that variable was missing by adding a column, which is true or false if it was missing
‫or not.
‫So that means if we find a row that has a missing auctioneer ID value and then we fill it with the median
‫of the auction idea column, there's going to be a column on the very end of our dataframe which says
‫true, which is basically zero or one, which tells us that, hey, we filled up this auctioneer ID
‫row, but originally that or this row was missing this value.
‫We're keeping that data.
‫We're keeping that information that the fact that that particular sample had a missing auctioneer ID
‫in the data set.
‫So we're keeping that information there, but then we're filling it up with the median.
‫And you might be asking why the median, right?
‫Why not the mean?
‫Well, the thing here, the reason being, is because median is more robust than the mean.
‫And you might be wondering, what do you mean that median is more robust than the mean?
‫Well, the thing is, when you use the mean, when we have a lot of different values, so we have 412,000
‫values, the main of a lot of different values can be very sensitive to outliers.
‫So let me demonstrate it with a kind of a rough example.
‫So demonstrate how median is more robust than main.
‫And we've just run this cell.
‫So we'll check out what happened with that in a second.
‫And I just want to give you an example of why we might use the median.
‫You might be on a search for this go y use median over main.
‫That could be some sort of extension.
‫You could do some research there but we go here and go tens equals NP fall.
‫What this is going to do.
‫We're creating an array a thousand long with a value of 100.
‫So say there's 1000 people.
‫Why are we calling it this tens?
‫So 100.
‫Hundreds.
‫There's a thousand people each with $100 each, and then we'll make one with hundreds and a billion.
‫Let's say out of those thousand people, Bill Gates just walked in and he's got $1,000,000,000 in his
‫pocket.
‫So we'll go not tens.
‫We want hundreds.
‫So there's a thousand people with $100.
‫But now Bill Gates has walked in and he has a billion.
‫That's nine zeros.
‫My goodness.
‫That's a lot of zeros.
‫One, two, three.
‫Well done.
‫Okay.
‫So what we're going to do here is go up main hundreds and then we're going to empty main hundreds billion
‫and then we're going to get empty median hundreds and then we're going to go in median hundreds billion.
‫So all we're doing is we're taking the mean and the mean of each of these and the median.
‫The median of each.
‫And let's see.
‫Whoa.
‫So see what I mean?
‫Here in our array?
‫Hundreds.
‫Let's have a look at it.
‫This is just a thousand examples of the number of 100.
‫The main is 100.
‫That makes sense.
‫And the median is also 100.
‫But when we add 1 billion to the end should be right down here somewhere.
‫Or maybe it's not.
‫It's in hundreds billion.
‫Now, we had one example of a billion being right at the end.
‫Look how much that influences our mean.
‫And now this is an extreme example, right?
‫We're using a billion versus 100.
‫So we've got a couple of orders of magnitude here.
‫Look how different it is to the median.
‫In its purest form.
‫This is why we're using the median.
‫And so you might come across that in your research of why using media to the main.
‫But the main reason is, is because the median is more robust to outliers.
‫But with that being covered, let's have a look at what we just did.
‫What did we just do with this?
‫So to figure that out, what we can do now is check if there's any null numeric values.
‫And how did we do that before?
‫Well, we did full label content and DF template items, and that's what we're after.
‫If PD dot, AP dot is on time, that is I'm getting ahead of myself here.
‫Numeric D type content.
‫Wonderful.
‫If PD is null content some print label.
‫So this should tell us if there is still any numeric values or numeric columns that have null values
‫in it.
‫It should print out those column names.
‫And it prints out nothing.
‫The reason being is because we just filled the content.
‫The ones which were missing with the median are.
‫Now, let's check to see what our binary column did.
‫Okay.
‫Check to see.
‫So if this worked, we should have.
‫Let's go up here.
‫How many auctioneer ID values we're missing?
‫Can we find that out?
‫From our previous one.
‫AUCTIONEER ID.
‫Okay, so we should have.
‫The fact that we filled up false should be the total number of rows minus this.
‫So the true should be this value of our binary column.
‫Right?
‫Because it wasn't missing.
‫So let's go here.
‫Oh no.
‫True.
‫Should be this number.
‫Because that's how many missing values they were see there.
‫So what we've done here before, we filled our values, we've checked to see out and missing values
‫there are.
‫So our binary column should be true for 20,136 examples, because that's how many examples we've filled.
‫So let's check that out.
‫So def underscore temp dot because we've created a column auctioneer.
‫ID is missing.
‫See here we've got label plus is missing.
‫Now this should be true and false.
‫Beautiful.
‫What that means is that we filled up 20,136 values in the auctioneer ID column.
‫Not there is missing column because remember, we just made this with the median of the original auction
‫idea column.
‫Who?
‫Now that was pretty full on, but we've still got some missing values in our dataframe.
‫Let's check out what else do we have?
‫We filled up all the numeric missing values, but we've still got some of these.
‫Hmm.
‫We still got a fair few.
‫All these ones are zero.
‫They don't have any missing values, so that's good.
‫They're getting ready to use with our machine learning model, but we still have to fill all of these
‫before we can use our data with a machine learning model.
‫So let's do that in the next video.