‫Okay.
‫You've made it.
‫Look how far we've come.
‫We've covered an absolutely massive chunk of the psychic loan library, and we're still going to finish
‫it off.
‫We're up to the last section, putting it all together.
‫We've been through an end to end workflow.
‫We've seen how to get the data ready.
‫We've seen how to choose the right machine learning model for our problems.
‫We fiddle it to the data.
‫We've used it to make predictions.
‫We've evaluated a trained model.
‫We've improved a trained model.
‫We've saved and loaded a train model.
‫Now let's combine all of this, all of this.
‫Could we do it?
‫Now, let's make a little heading 7.0, putting it all together.
‫First thing.
‫Give yourself a little pat on the back because we've covered a lot and so far it seems to be all over
‫the place and which it is.
‫But not to worry, this is how machine learning projects often start out.
‫There's a whole bunch of different experiments and code all over the place.
‫And then once you've found something which works, the refinement process begins.
‫And now what would this refinement process look like?
‫Well, let's use remember our good old car sales regression problem, a.k.a. predicting the sale price
‫of cars.
‫We'll use that as an example and to tidy things up, we're going to be using CI kit loans.
‫Let's have a look CI kit learn pipeline to tidy things up.
‫We're going to be using the pipeline class.
‫Now, you can imagine Pipeline as being a way to string together a number of different cyclic loan processes
‫in one hit.
‫So similar to like writing a function, right?
‫You're getting a couple of different steps and you're putting them together.
‫Pipeline of transformers with the finalist miner.
‫So Transformer or transform is IKEA transforming data, getting data ready.
‫And then a final estimate are estimated.
‫Remember in psychic loan terms is a machine learning model.
‫So as a refresh, this is our car sales data is 1000 rows and there's some missing data here.
‫Right?
‫So there's missing here.
‫Missing here.
‫And what we're trying to do is use make color odometer doors to predict the sale price of cars.
‫Now, again, this data is imperfect because could you do this?
‫Could you figure this out?
‫Given enough time, you'd probably realistically need a few more parameters, a few more details about
‫the cars before you could figure out how much they'd sell for.
‫So let's stop talking about it.
‫Let's get hands on with the code to to start off, we'll re-import the data.
‫So go pd read see as v.
‫Don't.
‫We've saved it in our data folder.
‫Car sales extended or no data folder is not is not up the chain.
‫I want carsales ex.
‫There we go.
‫Carsales extended data missing.
‫That's what we want.
‫And then we'll have a look.
‫We'll have a quick peek at it.
‫Wonderful.
‫So this is just reporting this.
‫CSV file that we have here.
‫Come back.
‫Now.
‫What did we say?
‫We said we want to use these columns.
‫So these four columns to predict the price column.
‫We'll have a look at our data types.
‫So we've got make, which is an object that's a string.
‫They're color which is an object.
‫That's a string there.
‫I don't want our doors in price, which are we've kind of talked about doors being a category as well.
‫So odometer is our numerical column.
‫And do you remember what we have to do for our data?
‫We have to make sure that it's all in numerical format before we can build a machine learning model
‫on it.
‫And there's another little roadblock that we have as well before we can build a machine learning model
‫and that's if we go is a so is null some we've got null values in some of our while all of our columns
‫we've got 50 missing missing data points in each of our columns.
‫So that's also a big no no with machine learning models, we need to make sure that the data is numerical
‫and that there's no missing values.
‫We refer back to our keynote.
‫All data should be numerical.
‫This is things to remember, right?
‫And there should be no missing values.
‫Okay.
‫Let's check it out.
‫So what we'll have to do is if we want to put this all together, we've done this before.
‫We've seen filling data, we've seen converting it.
‫But this time you want to use a pipeline to do so.
‫So the psychic learns.
‫Pipeline's main input is steps, which is a list that contains a tuple which has step name and action
‫to take.
‫We're going to see this in a second, but we're just talking through it.
‫And in our case, our steps are what are our steps?
‫Well, first, we need to fill the missing data.
‫And then convert the data to numbers and then build a machine learning model on it.
‫So let's do it.
‫We'll go here.
‫I want to do this one.
‫So it's all in one cell.
‫Let's put our steps here of what we have to do.
‫Steps.
‫We want to do all in one cell.
‫Can we do it all in one cell?
‫Because before to do this, it took a fair bit of work.
‫But let's see, this is what we're doing.
‫This is our refinement process using pipeline.
‫So step one is fill missing data.
‫And then step two is convert data to numbers.
‫Wonderful.
‫And then step three is build a model on the data.
‫All right.
‫Cracks, knuckles, gets fingers ready.
‫Let's do it.
‫And if it seems like I'm going through this pretty fast, let me tell you, there's nothing here that
‫we haven't covered except for the pipeline.
‫But what we've learned in the previous videos, you'll be able to deduce, as well as reading the documentation
‫here, what exactly is going on.
‫So import PANDAS is PD because we want to do this all in one cell and then from scalar compose.
‫Remember the column transformer?
‫And then from Skyline, this is where we're going to import the pipeline class that we're going to take
‫advantage of here, import pipeline.
‫And then from Skyline Compute, we're going to import simple and pewter.
‫This is what we're going to use to import our data.
‫So fill the missing data.
‫And then from skyline dot preprocessing.
‫Import one hot encoder.
‫This is what we're going to do to convert our data to numbers or sorry, we're going to use to convert
‫our data to number.
‫So getting data ready, these are the inputs we're going to use for that.
‫Then the next one is modeling.
‫So from Skyline Ensemble.
‫Import our trusty random forests regress us because we want it to predict a number.
‫And then from skylon model selection import.
‫We got that wrong.
‫Yeah, that looks wrong.
‫I've been known to be prone to typos.
‫Train test split.
‫And we want grid search CV because we've seen grid search being used before, which is going to search
‫for some optimal type of parameters for our random forest aggressor.
‫Okay, now we're going to set up random seed, so we'll import numpy as MP NP random seed 42.
‫Wonderful.
‫Now for completeness, for doing it all in one cell, we're going to re-import the data.
‫So import data and we're going to drop the rows with missing labels.
‫So data equals PD, not CSV.
‫We're going to go to our data folder.
‫This needs to be a string.
‫And I go car sales.
‫Extended missing data.
‫That's the one we're after the thousand rose version and then to drop the the rose that have missing
‫labels we'll go data dot drop and a subset will set that equal to the price column because that's what
‫we want to predict and that's our label and we're going to go in place equals true.
‫Okay so what we've done is we just replicated this here and now we've dropped.
‫The rows which contain missing price values.
‫So a.k.a. missing labels.
‫Now, what we're going to do here is we're going to start to define the different features of the data
‫set and the different transformer pipelines that we want to take place on that data.
‫So if we come back up here, what transformations do we need to make?
‫We need to change our making color into numbers.
‫We need to fill the missing values of all the columns in here.
‫And we need to change our doors because it's also a category.
‫We need to change it to a categorical value.
‫So we're going to actually know.
‫Forget what I said about doors.
‫Let's see what we're actually doing in the code.
‫So define different features and transformer pipelines.
‫So for our categorical features, doors, doors is a special case because it's already numerical.
‫Categorical features equals make.
‫And now what we're going to do here is create a categorical transformer.
‫So this just means modifying the data.
‫And to do so, we're going to use a pipeline.
‫This is the first time we're seeing pipeline, but the steps in the pipeline are going to be first.
‫We're going to fill with missing values with an importer.
‫So this is where the pipeline inputs take steps to take, right?
‫One after the other as a list of tuples.
‫So let's see this happening.
‫The name is Importer and we're going to remember this is the action to take.
‫So simple computer strategy equals constant and then the fill value is going to be missing.
‫So you might be able to deduce what's happening there with this particular case.
‫Where imputing.
‫The categorical features with the constant value of missing string.
‫Now we're going to create one hot and go one hot encoder.
‫And to handle the unknown.
‫So columns that hasn't seen before.
‫We're going to tell it to ignore them.
‫Wonderful.
‫And then we'll make sure that we've all lined up all lined up these things.
‫So we're going to need to we're missing something here.
‫I need to check where our brackets are at.
‫We're missing one here.
‫There we go.
‫See, half of this is figuring out where you're missing brackets are.
‫And then door feature.
‫We're going to define our door feature equals doors.
‫And so the same thing again.
‫So we're going to create a door transformer.
‫Door transformer equals pipeline.
‫Again, we're using a pipeline because this will all make sense in a second.
‫Right.
‫Why?
‫We're setting up pipelines.
‫So to do this, we have to press.
‫Impute to.
‫Now we want to do the same thing again.
‫Simple computer.
‫And the strategy is going to be constant and the full value for our doors is we're just going to fill
‫it out with fours because four is the majority number in the Doors column.
‫Then next we want numeric features.
‫So our numeric features is our odometer column.
‫Cam.
‫Wonderful.
‫And then we're going to create a numeric transformer equals.
‫First thing we're going to do is pipeline steps equal.
‫And then again, we need to impute values here.
‫So impute to.
‫Simple computer.
‫Strategy equals main.
‫So this means it's going to fill the numerical columns.
‫In our case, the odometer with the strategy means it's going to take the main value of the odometer
‫column and fill all of the missing rows in odometer with the mean of the rest of the values.
‫Whew.
‫That was a lot.
‫Now, still not done.
‫That's all right.
‫We're going to set up the preprocessing steps.
‫So remember what we need to do.
‫We need to fill missing data.
‫So this is part of our preprocessing.
‫These two are preprocessing our data.
‫And this is modeling the data.
‫So our preprocessing steps are going to be fill missing data and convert data to numbers.
‫So we can do that, fill missing values, then convert to numbers.
‫So to create a preprocessor, we'll instantiate a preprocessor using column transformer tab autocomplete,
‫and we're going to pass it.
‫Column transformer takes transformers equals.
‫And now this is going to be a list of the transformations we want to do on our data.
‫And in our case, we have one called Cat for short, for categorical.
‫It's going to pass it the categorical transformer.
‫And we want those transformations that we've defined up here, the categorical transformer to fill the
‫value simple computer with missing and to one hot and code make and color.
‫We want those to be performed on the categorical features.
‫Press tab water complete categorical features.
‫Wonderful.
‫Now we pass it another tap on.
‫So now we want it to do same steps, but this time using the door transformer.
‫Which is going to fill the door column with the constant value of four.
‫So we can go door feature.
‫Wonderful.
‫And finally, I wanted to finish up using the num transformer, which is going to take our numeric transformer.
‫And perform.
‫The imputation.
‫On the numeric features.
‫Beautiful when there's only one feature.
‫So features as a plural is probably probably not correct.
‫And now now that we've done our preprocessing, right, so our data is getting filled, the missing
‫values are getting filled, and then it's getting converted to numbers.
‫Using what we've done here.
‫The next step is to create a preprocessing and modeling pipeline.
‫We can do this as you maybe out of guess because I've used the word pipeline where you need to create
‫a preprocessing and modeling pipeline.
‫So now essentially what we're doing is we're putting our preprocessing steps together with a modeling
‫step, and because there's multiple steps will use a pipeline to combine them.
‫So model equals pipeline.
‫Pass it steps, which is a list containing tuples.
‫And this is the name.
‫So this is a preprocessor step.
‫And we're going to pass it preprocessor.
‫So the first step in our pipeline here, our model pipeline, is to run through this.
‫And then the next step.
‫Put a little comma here, and we need to create another tap on.
‫Which is going to be our model is our random forest aggressor.
‫Wonderful.
‫So now we've got our model pipeline set up, which will go through Preprocessing.
‫Once the preprocessing is done, it'll build a random forest classifier on it.
‫We can get our data ready.
‫So split data X equals data drop.
‫We're going to remove the price column because that's the one we want to predict.
‫The access is going to equal one.
‫Y is going to be data price.
‫Then we're going to create x train x test as we always do.
‫Y train y test equals train test, split x y test size, we'll use 0.2.
‫And then we're going to fit and score the model.
‫It's a model dolphin x train y train now model score.
‫X test.
‫Why test?
‫Oh.
‫Where we got it wrong.
‫Handle on known.
‫We got a typo.
‫Of course we do.
‫Handle unknown.
‫There we go.
‫Change that.
‫Whoa.
‫We did it.
‫We did the entire pipeline in one cell, taking advantage of the pipeline class.
‫So let's step through what we did.
‫We imported the classes to get our data ready.
‫Pandas column transformer pipeline symbol and period one hot encoder.
‫We imported the classes to do our modeling, so random forest repressor train test split grid search
‫CV.
‫We set up a random seed so our results are reproducible.
‫And then we define different features and transformer pipelines.
‫So we define the categorical features here, the door feature, the numeric feature.
‫And then for each of them we created a few steps for our pipeline to take to not only impute missing
‫values but to one hot encode.
‫So turn all of our data that wasn't numbers into numbers.
‫And then we set up a pre-processing step.
‫That took all of our pipelines here.
‫To take the steps that we wanted to take on the categorical features on the door feature and on the
‫numerical feature.
‫Then we created another pipeline to put all of these steps together.
‫To not only pre process our data and then model it.
‫So then we split our data into X and Y and then into train and test.
‫And so when we call model fit, it's going to call this variable here, which is going to run this pipeline.
‫Which inherently is going to call the preprocessor step, which is going to run this pipeline, which
‫is inherently going to run all of these three steps.
‫You know what I'd be thinking, Daniel, while this has got a lot going on, but if you go back through
‫what you've looked at here, you'll be able to say you can step through it and go, okay, I'm starting
‫to piece together what's happening.
‫And now if you scroll right back up to where we originally did this, filling data with pandas, filling
‫data with Skylon, and then modeling it on our car sales problem, you'll see that we did it over a
‫number of different cells and now we're doing it.
‫We're building a full blown machine learning model in 55 lines of code minus some comments, and we
‫could probably tidy this up if we're being honest with ourselves.
‫But now you're starting to see is in the beginning, machine learning is very experimental throughout
‫the whole thing.
‫It's very experimental.
‫And so what it takes at the start is just code everywhere, trying different things, seeing how they
‫work.
‫And then eventually you can bring it down to something like this, put it all into one big cell in a
‫pipeline, going through the steps that you've figured out along the way.
‫Now, of course, we'd want to probably improve this use hyper parameter tuning, and that's what we
‫might look at in the next video is how to use hyper parameter tuning with a pipeline slightly different
‫to how we've seen it done with just a regular model.
‫So take a little break, step back through what we've done here, because it was a lot.
‫Don't worry if you don't take it in entirely.
‫Haven't read through the pipeline.
‫Class of psychic loan documentation and see if you can line up what we've done in the documentation
‫to what we've done here.
‫Otherwise.
‫I'll see you in the next video.
‫We'll see if we can improve this score.