‫Well, we're making good progress on this.
‫There's a price regression problem.
‫In the last video, we tried to build a machine learning model.
‫We learned that we've got some data that isn't in numeric format.
‫So see here, we've got some strings and if we check.
‫DF Oh, actually we're working with DF Temp.
‫My bad C These are things that you're going to forget, that you're going to go through, that you're
‫going to mix up.
‫So DF Temp still same problem, right?
‫Because we've got some values here, some columns that are numeric and a whole bunch that aren't.
‫So what can we do about that?
‫Well, not even.
‫Daniel, what are you doing?
‫You're not even working with your temporary data frame.
‫Okay, well, this is all part of the game, right?
‫Making mistakes and then going back and fixing them.
‫That's part of being a data scientist.
‫Heaps of experimentation.
‫So one way we can turn all of our string data or non numeric data into numbers is by converting them
‫into PANDAS categories.
‫That's what's that mean?
‫So convert strings to categories.
‫Now I'll put it here.
‫One way we can turn all of our data into numbers is by converting them into PANDAS categories.
‫So like the datetime object that we saw when we imported sale date, PANDAS has another data type called
‫category.
‫Let's have a look at the Pandas types API.
‫Well, there we go.
‫This is probably a bit better.
‫We can see this here.
‫So pandas dot API types.
‫Where can we have a look at this?
‫Utility functions, data types, related functionality.
‫So here we go.
‫Let's copy this, but let's just read it out first.
‫What's going on here?
‫D type introspection.
‫So D types is ball type is categorical type.
‫Okay, pandas d type convert input into a pandas only d type object on numpy data type object.
‫Okay.
‫So we've got a fair few options here.
‫To have a look at different data types with pandas.
‫But since we're focusing on converting strings to categories, let's see how we do that.
‫Let's put a little link here.
‫We can check the different data types compatible with pandas here.
‫So remember, a lot of machine learning is basically getting some data, what we got here and then preparing
‫it to run with an existing machine learning model that we've chosen from our machine learning map isn't
‫here.
‫So it's about getting some inputs.
‫Okay.
‫A whole bunch of data that we've got here.
‫All this sort of stuff here and then massaging it or essentially, yeah, that's a good word for it.
‫Massaging it into a way that it works with a machine learning model.
‫So that machine learning model can find the pattern.
‫Someone else has built this model for us and we're going to thank them very much.
‫And then it's going to find the patterns and give us some outputs.
‫Okay, some predictions, some patterns, some labels, something like that.
‫And then we can evaluate how well that's works is what we're doing here.
‫We're manipulating our inputs to work with our machine learning model.
‫So let's have a look.
‫Let's remind ourselves of temp head.
‫Of what we're working with.
‫Okay.
‫Yeah, we've seen this a fair few times.
‫This is just another way to look at it.
‫Always a good idea to look at your data.
‫And so what we might do is try to use one of these API types.
‫So what can we use?
‫Is there a string one here?
‫Here we go.
‫IP types is string D type.
‫How do we use this is string D type string.
‫Okay.
‫Let's try this out.
‫So paid API types.
‫It is string d type def temp.
‫Usage banned use of warning is there?
‫Why is there a use warning?
‫Is there not to use a warning alarm?
‫I don't know.
‫Who knows?
‫I'm going to keep making that mistake, but let's check.
‫Okay.
‫So it is a string.
‫The usage band column is a string.
‫Now we could do the same thing for almost every other column, but why would we go through and write
‫that much code?
‫Let's build a little for loop.
‫That does it for us.
‫So fine.
‫The columns which contain strings.
‫Because if we know which columns contain strings, then we can convert them into some kind of number
‫and our machine learning model will be happy.
‫So four label columns in Def Temp.
‫Now I'm going to go through this 1/1 and then we'll retrospect it afterwards because you might be looking
‫at this going, I don't know what DF temp items is, but maybe you want to try it out.
‫Maybe you want to pause a video and try it out and you explain it to me, not types.
‫And if not, that's all right.
‫We're going to go through it D time now.
‫All we're doing is we're just copying something like this because we're trying to find all the columns
‫with strings and we're going to pass this.
‫Or actually this is more content.
‫That's what the documentation says.
‫And we're going to pass content here.
‫So this makes sense.
‫You don't know what the items DF items does unless you've checked it, but we're going to go for label
‫content and then we're going to check if the content is a string data type.
‫Yes.
‫Then we're going to print the label.
‫Now, you might be able to guess what this does, but if not, that's okay.
‫We're going to see what does anyway.
‫If in doubt, run the code and we got here.
‫So these are all the columns that are string data types.
‫Does that make sense?
‫If you're wondering what DF temp items does?
‫It treats our dataframe like a dictionary, which is kind of what it is, right?
‫It's a dictionary of keys and values.
‫Okay.
‫The keys of the column names and the values are what you see in the in the column.
‫So let's give a little example.
‫If you're wondering what default items.
‫Does.
‫Here's an example.
‫So we'll create a random dict dictionary equals let's be creative key one equals hello and a two equals.
‫Well, maybe you can be a little bit more creative than me, but we're going to go for key value in
‫random dick items.
‫So see how we've got just our items.
‫Same with up here.
‫Dot items.
‫We're going to go.
‫We want print if this is a key, and then we'll pass it.
‫And then we'll also go.
‫If this is a value.
‫Value.
‫Wonderful.
‫So if you work with Python dictionaries before, you might be able to tell what's going to happen here.
‫What have we got here?
‫Oh, I see.
‫It's printed out twice.
‫That's alright.
‫We can kind of get what's going on there.
‫So essentially what happens here is for labeled content is basically labels the column names and the
‫content is the content of the column.
‫So this is what this is doing.
‫This is this brilliant little for loop tells us all the columns that contains the string data type.
‫Wonderful.
‫So now that we know that, what can we do?
‫Hmm.
‫Well, if we were to read through our pandas data types related functionality.
‫Right, we're not going to do that.
‫But it's somewhere in here or somewhere in there.
‫Or you could also search up how to change the data type of a pandas column.
‫One of the things you might stumble across with is converting strings into categories rather than talk
‫about it.
‫Daniel, come on.
‫Let's just.
‫Let's just code it.
‫Let's see it happen.
‫All right, all right, all right, let's do it.
‫So this will turn all of the string values into category values.
‫And you're wondering, why would we do this?
‫Well, it's going to make a lot of sense in a second.
‫So for label, we're going to just bring the same for loop that we wrote up before DF temp items.
‫And we're going to go if the peed on API this is a fairly long one types is string D type that's a mouthful
‫so basically is the content of string.
‫If true we want to def temp.
‫We want to change that column.
‫So see here the column name.
‫So we want to access that column name.
‫We want to set it equal to the content.
‫So write keep the column name but change the content to as type category.
‫I need to spell this right.
‫Dot cat.
‫As ordered.
‫Oh, my goodness.
‫What is happening here?
‫Well, let's have a look.
‫What is going to happen?
‫Hmm.
‫This might take a little while because we've got to go through a lot of columns, but we're going to
‫preload our next Jupiter cell, right?
‫Because we're pumping through this beautiful DF temp dot info.
‫Have a look here.
‫What is going on?
‫What have we changed?
‫I'll give you a hint at something over here.
‫Well, it is this.
‫So now these used to be objects, okay?
‫They used to be strings.
‫Now we've changed them into categories.
‫Hmm.
‫Well, that's interesting.
‫What does this mean?
‫Well.
‫Aha.
‫This is where we can have a look.
‫Remember, we're trying to make sure all of our dataframe is numeric so we can't pass strings to a machine
‫learning model.
‫So this is what we're working towards, a DF temp.
‫Let's check out which one did we check before?
‫Was it the state?
‫Beautiful.
‫Let's check that.
‫So def temp dot state not cat categories.
‫Wonderful.
‫There's Old Faithful Wyoming.
‫Shout out to Wyoming and every other state, wherever you're from and every other country, to be honest,
‫because this is on the Internet.
‫All right.
‫So we have categories here.
‫And what this is going to do is remember when we did the state value counts.
‫Remember when we did this?
‫So if we were to go through all of these because remember how we did up here, there's a fair few things
‫we've just covered here.
‫Then you'll come on, explain them as ordered means.
‫These are now ordered.
‫See how the alphabetical.
‫Aye aye.
‫Aye, aye, aye.
‫See?
‫Be all right.
‫And there's no be.
‫These aren't ordered here while they're ordered by their value counts.
‫But essentially what this has done is it turned it from strings to categories.
‫So this is going to assign when we called this.
‫It's going to assign a numerical value to each category.
‫So now looking at this, it doesn't really make sense, right?
‫Because these are still look like strings.
‫However, under the hood, pandas is treating these as numbers.
‫So Alabama might be one, Alaska might be two, Arizona might be three.
‫And we can check that by going def temp, dot state dot cat four category.
‫So we're accessing the cat.
‫Remember how we access the date time using date?
‫Well, now we're accessing the category barcodes.
‫Okay, so now we have a way that our data frame can be accessed.
‫At least all the string values right.
‫Have been now converted to categories because see here we loop through our data frame and we figured
‫out whether they're a string or not.
‫And we've changed their data type to categories and we've given them order.
‫So now we have all these categories here and we have a way to access their numeric value.
‫Who?
‫Not that.
‫Hey, I'm rubbing my hands together.
‫You might not be able to hear that because I'm really excited.
‫But even though we have a way to access our data as numeric, we still have a whole bunch of missing
‫values.
‫So let's write this down.
‫Let's give some thanks.
‫So, thanks to pandas.
‫Categories.
‫We now have a way to access all of our data in the form of numbers.
‫Beautiful.
‫But we still have a bunch of missing data.
‫Let's have a look at that check.
‫Missing data.
‫So we've got DF temp dot is now.
‫Got some.
‫Now let's get some ratios here.
‫Right.
‫So let's divide it by the length of dataframe.
‫So we've got nine amazing values in these columns here because these are all zeros and then we have.
‫Okay, so not much missing there, but usage band has 80% of missing data.
‫85% missing data, 81% missing data, 52%.
‫So all 93 my goodness, maybe something's going on there.
‫We'd probably have to.
‫If this was a data set in the wild we were working on with the client, maybe we'd ask them.
‫But since it's from Kaggle, accessing a subject matter expert is probably pretty hard on this dataset.
‫So these are some things we might explore a bit more, but we're on our mission to build a machine learning
‫model.
‫So what we might have to do next is fill these missing values.
‫But before we do that, since we've made some changes on our DF temp, we'll save it to a new CSV so
‫we can.
‫If we were to just start this notebook from Fresh, we can just import our manipulated data frame and
‫start exactly where we are now.
‫So save pre process data.
‫And we go export current temp dataframe, so we'll go diff temp to CSV, we'll put it back in the data
‫folder.
‫So Blue book.
‫Playbook for bulldozers.
‫And we're going to call it just something simple like train temp, not CSB.
‫Beautiful.
‫I want to index equals false.
‫Otherwise, we'll come up with some funny things when we re-import it.
‫And then we're going to import.
‫Pre processed data.
‫And now you could do this for whatever stage you're at, right?
‫If you've made some changes to the temporary data frame.
‫I just decided to do it now because it crossed my mind.
‫You can do it whatever stage you're at.
‫If you've updated some columns, if you've done some feature engineering, you can export it and re-import
‫it that way.
‫If you're coming back at a later session, you don't have to make those changes again.
‫You don't have to run your notebook from the top.
‫You could just run it from here.
‫Oh, no, we've just written it to file.
‫Daniel, come on, we want to read it now.
‫Page eight.
‫I'd say it's very.
‫This should work.
‫They should re-import it.
‫And then if we want def temp head, we should have the exact same data frame that we're working with
‫now.
‫Wonderful.
‫And so, again, we'll just remind ourselves that we have a lot of missing values to deal with.
‫Beautiful.
‫Just because we export it and re-import it out, DataFrame doesn't mean that our missing values have
‫been handled.
‫So that's probably what we're going to do.
‫Well, it is what we're going to do in the next video as we work towards getting our data ready for
‫a machine learning model, let's start to fill some missing values.