‫Hey, it's Andre here with a quick little tip for you.
‫Now, over the next couple of videos, Daniel is going to show you his machine learning workflow.
‫And one of the things that he's going to do is, well, clean the data, transform it, and do all sorts
‫of manipulation to it.
‫Now, I want us to take a high level view right now, because when we're working with Jupyter Notebooks,
‫it can get really confusing.
‫It almost seems like we're doing a bunch of steps without understanding the why.
‫But I want to take a step back here and understand why we do the things we're going to do over the next
‫couple of videos you see as a data scientist.
‫More data doesn't necessarily mean it's always good.
‫I mean, yes, generally the more data we have, the more data we have to work with.
‫But at the end of the day, we can have completely useless data.
‫We only want useful data so that we can get real results, real actionable steps that we can take.
‫And what we're going to see Danielle do is do this step before we even create our machine learning model.
‫You see, we want to run this step.
‫Cleaning data, transforming data and reducing data.
‫So let's talk about that.
‫Why do we need to clean data?
‫Well, we want to remove and replace data.
‫Because sometimes data is missing, right?
‫We might have tables with missing values, missing labels.
‫That's not going to help us and it's not going to allow us to build the right machine learning model.
‫So we usually remove a row or a column that's empty or has missing fields, or we might calculate some
‫sort of average.
‫If we're doing, house prices will fill an empty price list to maybe an average house.
‫You might even notice some outliers in your data and just remove them.
‫Next is data transformation.
‫You see, computers don't really understand concepts, right?
‫Most of the time, computers just understand ones and zeros.
‫Just numbers, really, in order for us to teach computers.
‫Most of the time, we want to transform our data into some sort of form that a computer can understand.
‫So what we're going to see is Daniel actually convert some of our information into numbers.
‫You might have things like the price of a house or a salary that are numbers and easy for us to create
‫models with because computers understand numbers fairly easily.
‫But it might find it hard to know what a color is.
‫What is green?
‫What is blue?
‫A computer doesn't understand that.
‫Instead, we convert colors into numbers like our gbps colors so that a computer understands.
‫A good way we transform data is usually between zeros and ones such as Do you have heart disease or
‫do you not know?
‫Heart disease is zero.
‫Heart disease is one.
‫We have to make sure that data across the board uses the same units.
‫Maybe one data uses meters, another data uses yards.
‫This idea of transforming data into a useful form is really, really difficult and it just comes with
‫practice.
‫But you're going to see Daniel how he transforms data so it's more and more workable.
‫So once we've cleaned the data, once we've transformed the data, next is to reduce our data.
‫Why reduce our data?
‫Don't we want as much data as possible?
‫Well, theoretically, yes.
‫But remember, everything costs money.
‫The more data we have, the more CPU, the more energy it takes for us to run our computation.
‫So if we're able to get the same result on less data, that actually saves us money.
‫It saves companies money, especially big companies like Google that have a lot of data and run a lot
‫of CPU.
‫Sometimes the idea of data reduction can also be called dimensionality reduction.
‫Or column reduction.
‫You can have many columns on your table, but it makes things really, really slow.
‫So you can actually remove some columns that maybe you find irrelevant and won't be needed for your
‫machine learning model.
‫Again, these are things that Daniel is going to go through, but I want you to keep this in mind,
‫this idea that as a data scientist, you can't just assume all data you have is automatically going
‫to be perfect.
‫You have to clean it.
‫You have to transform it.
‫You have to reduce it.
‫This idea of wrangling data, of moving around data into a form that is useful to you is part of the
‫skill.
‫So watch Daniel do that and learn along the way.
‫I'll see you in the next one.
‫Buh bye.