‫Now we filled up all the missing numerical values in our data frame.
‫Let's do the same.
‫But for the rest of the missing values, which are these ones here with numbers?
‫So we found this out.
‫Using DF temp isn't a sum, so we've got a fair few that are still missing and I have a hunch that they're
‫all categorical.
‫Remember how we found columns before that with the string data type?
‫Let's do that again.
‫But this time, because we've filled up all the numeric data, let's find all the columns that aren't
‫numeric data.
‫And that should tell us all of these columns that still have missing values.
‫Does that make sense?
‫So because we've filled up all of the columns with numeric data, we want to find the columns that aren't
‫numeric data.
‫So make a little heading here.
‫So filling and turning categorical variables into numbers.
‫We're working on categorical variables now because we've already worked on our numerical variables.
‫So let's go check check for columns which aren't numeric and we'll go for label content and def temp
‫dot items and then we'll go if not PD not API types, dot is numeric.
‫So this is make sense.
‫If not so if they're not numeric because we filled up all the variables which are numeric.
‫Now we want to find the ones which aren't numeric content and then we want to go print label.
‫Beautiful.
‫Now, this is going to tell us all of the values, all of the columns that aren't numeric, but some
‫of them.
‫Okay, usage band that has a fair few missing value.
‫So that has 339,000 missing values.
‫So now we know all of the columns that aren't numeric.
‫What can we do?
‫How do we fill these up?
‫When we turned a string column into categories, we had access to an attribute called Codes, which
‫gave us a numerical value for all of the variables in that column.
‫Let's have a look at an example before we do it.
‫So PD Categorical.
‫So what this is going to do is go panda's categorical type and then we're going to pass it DF Temp State.
‫We'll look at the state column because that's the one we use an example before and we'll see what this
‫does.
‫Categories.
‫53 objects.
‫So this is categories.
‫Okay.
‫Can we do d-type here?
‫Categorical D-type.
‫Okay.
‫Yeah.
‫And what might do is codes.
‫Uh, okay.
‫So this is how we turn all of the state.
‫The variables in the state column into numbers.
‫So how would we do this for everything?
‫All of the categorical columns.
‫Because that's what we need to do, right?
‫We need to fill in the missing categorical variables and turn them into numbers at the same time.
‫So we might use this.
‫Let's keep that there.
‫I'll make another cell here.
‫We might go turn categorical variables into numbers and fill missing.
‫So let's say let's loop through for content in DF temp items and then if not so if it's not just like
‫we've done up here.
‫So if it's not a numerical type PD, not API times, it is numeric type.
‫But remember we've got the not here, so it's checking if it's not numeric daytime content.
‫Wonderful.
‫Because we know it's not numeric.
‫We might check to see if it's missing again.
‫And we're going to add a binary column just like we did in the previous video with numerical data.
‫So add a binary column to indicate whether the sample had missing value.
‫Wonderful ZDF temp.
‫Here we go label plus we'll do a little string hit is underscore missing equals paid is null content
‫so that's just going to return true or false if the value was missing.
‫So if the categorical value is missing, it'll return true.
‫It'll make a column and it'll assign that to true.
‫If it's not missing, it'll just have false a.k.a. true as one.
‫False is zero turn categories into numbers and add plus one.
‫You might be wondering, Daniel, why are we adding plus one?
‫Well, actually, let's write the line of code first because let's see the code paid categorical content
‫code plus one.
‫Now, I'd be wondering why are we doing plus one to all these codes?
‫So if we did this plus one, see how it goes up.
‫Keep that there goes up.
‫Okay.
‫Why would we do this?
‫Well, the way PANDAS categories works is it automatically if there was some missing value in here,
‫it assigns that a code of negative one.
‫But we don't want that.
‫We want it to be not negative one.
‫We want it to be zero.
‫So maybe if we find a column that has missing values, we might be able to see this.
‫If not, we go usage.
‫Oh no.
‫Is it usage bend.
‫That one has a lot of missing values, doesn't it.
‫Usage ban.
‫Okay, let's have a look at this column.
‫DOT codes.
‫Oh, negative one, negative one, negative one.
‫Beautiful.
‫We don't want this to be negative one, so we're going to add one there.
‫So all of our numbers are positive in our DataFrame.
‫That's the reason why we add plus one here.
‫So if we were to do this that way, we know when we turn our dataframe numeric into all numbers, all
‫categories.
‫That way we know zero is actually missing.
‫All the other categories are just plus one on what their category code is.
‫So let's.
‫Run this cell.
‫This might take a little while.
‫I have a fair few missing values, so we have to fill all of these.
‫So what we're doing, we're turning all of these columns that are non numeric, remember, that's what
‫we found here, not numeric.
‫We're taking those missing values or actually first we're turning them into categories.
‫If they have a missing value, that row will be assigned negative one, but we're adding plus one to
‫the code.
‫So all the missing values are going to be zero and all of the values that are still there, the categories
‫are going to be some number pertaining to whatever pandas decided that their categories are.
‫Oh, beautiful.
‫It's worked.
‫Okay, so now we can check to see what we've done by going DF temp info.
‫So we say, okay, so now we have 103 different columns.
‫Wow.
‫Well, let's have a look.
‫So if we go def temp dot head not t.
‫Ah.
‫Okay, so see, these are all our categorical is missing columns that have been adjusted that have been
‫added on the end.
‫Now this is from this line of code here.
‫So see what I mean?
‫Add a binary column to indicate whether sample had missing value.
‫So this means that for sample zero it had a differential type, it was missing.
‫So if we went back and found that column for sample zero, it would be filled with a value zero instead
‫of just missing.
‫Now.
‫One last check is to see if there are any more missing values, because what we've done filling up the
‫numeric and categorical values of our dataframe and turning them into numbers.
‫There shouldn't be any more missing values.
‫Beautiful.
‫So these should be all zero.
‫So maybe.
‫Is that getting a bit cut off?
‫Ten.
‫See what I mean?
‫It's getting truncated.
‫20.
‫Wonderful.
‫Sorry.
‫We've got no more missing values in our data frame, and our data should be all numeric.
‫What is that telling you?
‫What do you know about machine learning models?
‫What do those two points hint to you when thinking about machine learning models?
‫We'll wait to the next video so we can find out.