‫Okay.
‫Now we've made a copy of our original dataframe.
‫We can start to edit DF temp without any worries of damaging our original data.
‫So if we ever needed to revert back to it, we can with no troubles at all.
‫So what we're going to do in this video is we're going to take these dates, we're going to use pandas
‫date time, as well as some pandas date time attributes to enrich our data set.
‫And if you're wondering what is Dan you're talking about enrich our data set?
‫Well, there's little practice that we're about to dive into is called feature engineering.
‫If you remember right back up in step four.
‫Well, we've covered a lot already.
‫We've got a whole bunch of features and these are all listed here.
‫What feature engineering means is either creating a new feature.
‫So something like usage band may be completely different to usage band, but we would just add it on
‫here and then there could be a description or we would dive into one of these and go sail date and we
‫change it around a little bit.
‫So that's the process of feature engineering, so let's do that.
‫But with the dates now, how can we do that?
‫Well, first of all, we're going to see it in action.
‫Then we're going to discuss what's happening.
‫This is something you might want to do for almost any time.
‫Series project you're working on.
‫So let's make a little heading here.
‫Add date time parameters for sale date column.
‫Wonderful.
‫And Pandas has a beautiful little pandas date time.
‫Well, I just did pandas data.
‫I'm getting trigger happy date time attributes.
‫This is what we're going to be using, right?
‫We've turned our sale date column into a date time index.
‫See, they're immutable and the array of date time data, what we can do now is access some attributes
‫of that date.
‫And this is what I mean here.
‫So we can go year, the year of that date time, the month, the day, the hour, the minute, the second,
‫the microsecond nano second far out.
‫That's that's getting pretty deep.
‫Is it the start of the month?
‫Is it the end of a month?
‫Is it the start of a quarter?
‫Now, I want you to think about why might this value these features be valuable just without even seeing
‫it, without even coding it up?
‫If you're making a sale, why would it be a good idea to maybe know whether or not it was the end of
‫a quarter or the start of a quarter or the start of a month or something like that?
‫Why would it be?
‫Now.
‫It's okay if you're unsure.
‫But this is part of being a data scientist or a machine learning engineer.
‫Right?
‫It's about knowing a little bit about everything.
‫So if you're working on time series sales, maybe it's a good idea to know whether it's the start of
‫a quarter because sales might be increasing because it's the start of a quarter or maybe it's the end
‫of a quarter and sales are going down because everyone's tight on budget and maybe if it's the start
‫of the year, no one wants to buy anything, but it's the end of the year.
‫And people are like, Oh yes, let's, let's use the rest of our budget, let's buy heaps of things.
‫You don't have to know in-depth knowledge.
‫This is just one thing you can do to enrich your datetime objects.
‫So let's go in here.
‫How can we do this?
‫We can add a column.
‫What we're going to do is add a date time parameter for the sale date column on the end of our DF temp.
‫And now again talking about it isn't as good as having a look.
‫So we're going to say, oh yea, and how would we do this?
‫So what we're doing here is we're making a column DF temp sale year equals DF temp sale date.
‫So this means we're accessing the sale date column and then we can access the date time part of sale
‫date by using d t so d t date time and then we can access the year attribute.
‫By putting.
‫Year.
‫So let's have a look at this.
‫What does this tell us?
‫DateTime index dot year.
‫The year of the daytime.
‫Before we even reassign a column, let's just take the first row and see what this does.
‫1989.
‫Okay.
‫Now, let's do the same again.
‫But let's do it with DE.
‫Ah.
‫Okay.
‫17.
‫All right.
‫And now if we just did it with nothing.
‫So if we just did the sale date.
‫Oh, there we go.
‫So we see that that's accessed the year, that's accessed the day.
‫So let's enrich our dataframe with a few columns pertaining to these different values.
‫And if you're not sure what I mean, let's, let's see in an action, if in doubt run the code say oh
‫year equals that.
‫And then what we might do is we might just do month, day, day of week, day of year, and of course
‫we could do more.
‫But in the essence of time, really, we can make a function that does basically all of these.
‫But what we're going to do is just do five or six to demonstrate.
‫So let's do that.
‫We could copy it, but we're going to practice coding it out.
‫So we go here.
‫Sale month equals def temp sale date tab order complete d month and def temp sale day equals def temp
‫not sale date dty day def temp and we're going to go sale day of week.
‫That was when we said day of week.
‫Can you read that?
‫Maybe we do.
‫Camel case beautiful equals def temp dot sale date dot dot dot day of week def temp sale date of year
‫equals def temp.
‫Now, as you can see, there's a lot of redundant code here.
‫There might be a function in there somewhere.
‫That's a good challenge for you is to see if you can function like this and maybe even go to the PANDAS
‫documentation and add a few more different parameters to it to enrich our data set.
‫And now what do you think that this might do?
‫Well, let's have a look.
‫We'll run that and then we'll go here.
‫Def temp.
‫We're going to view the head and we'll transpose it so we can see all the columns.
‫Let's go right down the bottom.
‫There we go.
‫So what we've done is we've created.
‫So see, here is our year, our months, our day sale day of week.
‫It's our day of year.
‫We've created these five columns.
‫And what it's done is it just taken the sale date and extracted some of those values and added them
‫as more features on the back end of our dataframe.
‫Now, of course, these features won't be listed in our data dictionary, but if we were to go in and
‫share this with someone else, maybe that would be something we add to the data dictionary.
‫But we want to do that for now.
‫And so with this information here, we probably don't need the sale date column anymore.
‫This one here?
‫Probably a bit excess now.
‫So what we might do is remove that.
‫So let's drop it.
‫So we'll put in here little comment.
‫Now we've enriched our dataframe and remember because we're working on DF, but we have a need of the
‫Saoudite column back.
‫We could just go back to our DF our original dataframe that we we imported.
‫Now I've enriched our dataframe with date time features we can remove saoudite so let's go df temp dot
‫drop and we're going to go sale date on axis equals one in place equals true.
‫Remember in place happens in place so we don't have to go df ten equals.
‫Wonderful.
‫Now, if we try to do this, we're going to get an error because Saoudite doesn't exist anymore.
‫But that is okay because we've got the information.
‫It's just now across five different columns.
‫Boom, boom, boom, boom, boom.
‫And if we ever wanted it back, we can go back to our original dataframe.
‫Now.
‫I also like to check the values of different columns.
‫Now this is all about EDA write and exploratory data analysis.
‫It's just really checking out what's happening so what we might do.
‫So let's pick one.
‫What have we got here?
‫How about state?
‫US state in which sale occurred.
‫Wonder which state has the most sales?
‫If you're from the US, whatever state you're in, do you reckon.
‫Do you reckon you've made the most sales?
‫Let's have a look.
‫Value counts.
‫Maybe it's going to be Wyoming.
‫No.
‫Florida.
‫Texas.
‫A lot of a lot of sales are happening.
‫Wow.
‫Okay.
‫So Washington does.
‫Why even have a single sale?
‫I can't see it in that list.
‫Maybe you can.
‫Oh, there we go.
‫Wyoming.
‫Oh.
‫1672.
‫Okay.
‫So this is another part of the.
‫You could check more columns if you want.
‫You can go through and check different value counts.
‫Remember, value counts is really handy.
‫This is you're just looking at the spread of data.
‫That's another way of looking at the distribution.
‫I won't be tired of of exploring the data.
‫Let's do modelling.
‫Yeah, let's do that.
‫That's way more fun building machine learning model.
‫So you've done enough, Ada?
‫It could always do more, but let's start to do some model driven ADA Now what is model driven ADA?
‫Well, because there's so many features here.
‫We could spend all day just going through each of these columns, finding the value counts, doing a
‫distribution plot, making more plots.
‫But at the end of the day, we kind of know what our valuation metric is.
‫So why don't we try and just head towards that?
‫And if we can, well, how would we build a model?
‫Let's build a machine learning model.
‫But remember, what are we trying to do?
‫Well, if we come back, let's check DataFrame.
‫We're trying to use all of the independent variables to predict sale price.
‫All right.
‫So that means if we just drop all the columns, drop sale price, that should be our X.
‫And then if we use sale price, the column itself, that should be our Y.
‫But what kind of machine learning model do we use?
‫Well, remember, we've got psychic loan machine learning map.
‫If we go here choosing the right estimate.
‫And this is our old faithful.
‫So if we were to start here, if we go through this, do we have above 50 samples?
‫Yes.
‫Are we predicting category?
‫No.
‫Predicting a quantity?
‫Yes, we are predicting a number.
‫We are doing a regression problem.
‫Now, if we were to follow this through, we've got a bunch of different model options here and the
‫green.
‫But because we already had some experience with random forest, a random forest classifier, that is
‫what we might do is check out its counterpart, the random forest regress.
‫So if we were to go through here, forest of randomized trees, there's a classifier and there's random
‫forest regression.
‫All right.
‫Well, let's see what we can do.
‫We've seen this before with psychic learn of how to import a model.
‫So if we go from scalar dot ensemble and remember, if you want to see exactly how to do this, you
‫could just go into here random forest aggressor and it'll tell you what to do.
‫There might even be an example down here.
‫Now we go from Skyline Dot ensemble import, random forest regression.
‫That's what we're going to do.
‫Import random forest regrets.
‫Paul, this is exciting.
‫We're getting into modeling already.
‫We'll instantiate a model.
‫So model equals random forest regress and we're going to set the number of jobs to equal negative one.
‫What does this mean?
‫Well, that means it's going to use as many cores as possible on my computer because we're dealing with
‫how many rows are we dealing with?
‫Def temp.
‫We want the land.
‫We're dealing with 412,000 rose.
‫So that means our machine learning model has to go through 412,000 plus rows of this, try to work out
‫all the patterns between these columns and then try and calculate sale price.
‫That's a lot.
‫And what we might also do from now on is remember how we used to set.
‫We could do this.
‫MP Random seed 42.
‫We could set that in all of our cells.
‫But another way to do it is with our models.
‫There's a, there's a parameter called random state.
‫We can go Eagles 42 and remember to check the documentation of what's going on.
‫Oh, have we not are.
‫I can't press shift tab for some reason.
‫I'll just go to the documentation here.
‫We'll see what random state means.
‫Here we go.
‫Controls both the randomness of the bootstrapping of the samples when used, when building trees, if
‫bootstrapping was true, and the sampling of the features to consider.
‫Okay.
‫So long story short, this is going to.
‫It acts like np random seed and if we go here and jobs there we go.
‫None means one unless job leb negative one means using all processes.
‫So these are the two that we're setting, right?
‫Wonderful.
‫And so to fit it, we would go fit and we need to drop the sale price column.
‫So this is our X.
‫So sale price and axis equals one.
‫So we're dropping sale price and then we're going to pass it just the sale price column as the target.
‫So this is modeled fit.
‫This is X and this is why so def tamp sale price.
‫Let's see if it works.
‫Oh, no.
‫We've got a value error.
‫What's it saying?
‫Could not convert string to float low.
‫Ah.
‫You know why this is?
‫Is because if we go dfki info.
‫All of our data is not in numeric format.
‫Ah.
‫INT float int object usage band.
‫Okay.
‫What what type is that?
‫Let's have a look.
‫This is part of model driven EDA, right?
‫Our machine learning model just told us that thanks to the beautiful people of Psychic Line, they've
‫put in these errors to help us out of what's going on.
‫So usage.
‫Uh, no, not use warning usage band dot d type.
‫I don't know what's going on that d-type.
‫Oh.
‫Hmm.
‫Object.
‫So maybe it's a string or something like that.
‫And the other thing is, I just remembered we've got a whole bunch of missing values.
‫Hmm.
‫So from what we know about our machine learning models, we're trying to get into modeling, but we've
‫got data that's not numeric and a whole bunch of data that's missing.
‫So before we build a machine learning model, we'll have to take care of that.
‫So that's probably what we'll do over the next couple of videos.
‫We'll do some model driven EDA.
‫And the beautiful thing is, is we just tried to run that machine learning model and it told us, Hey,
‫we can't it's not going to work because you've got non numbers in your data, you've got some strings
‫in your data and if we were to go again it might tell us eventually that we've got missing data.
‫So that's what we need to handle first.