‫Okay.
‫So in the previous video, we understood that a data engineer essentially creates this data pipeline
‫for us, for people like data scientists to have available data to work with.
‫But I want to use a different example.
‫Let's pretend that Keiko Corp. has a need, and that need is for water.
‫The employees need water.
‫To work hard, right?
‫So let's think of this as a problem that a data engineer solves.
‫Now, bear with me.
‫We're going to be talking about water for a few minutes, but I promise it relates to data.
‫You see, the very first thing for us to get water is rain rate.
‫We need rain.
‫And this is similar to the data sources that we had.
‫That is the websites, the mobile phones, the cameras, the cars, the drones.
‫All of these are collecting data.
‫That's what the raindrops are.
‫So in order for us to get water to our employees, we have to get water from somewhere.
‫And that water comes from rain.
‫Next that water gets collected into different streams and rivers from all over the place, from all
‫over the world, just like data from all different places.
‫These streams of water start forming.
‫And what happens next?
‫Well, these streams of water turn into lakes.
‫They all flow into some sort of a lake.
‫And that lake kind of sits still.
‫But again, we as a corp just want water from employees.
‫Well, we obviously have a dam on a lake and that dam on a lake or a system to use that lake.
‫Well, we transport that water to something like a filtration sanitary area so that we can clean the
‫water, sanitize it, so it's safe for people to drink.
‫So we process that water that has been collected.
‫And then finally, after all of that, we create all the plumbing and pipes for us to deliver water
‫to Keiko Corp..
‫That's how we satisfy the needs of KeyCorp to provide water for their employees.
‫And this is how data works.
‫And this is what a data engineer builds.
‫A data engineer.
‫Starts off with what we call data ingestion.
‫That is acquiring data from various sources.
‫And we acquire all these different sources of data and ingest it into what we call a data lake.
‫A data lake is a collection of all this data into one location.
‫From there, we could just leave the lake as it is.
‫But there's a problem.
‫We don't want the lake to overflow, dry up.
‫We need to manage this lake somehow.
‫We don't want to flood the village.
‫We need dams.
‫We need pipes to the houses.
‫We need to filter and clean the water and we need piping to the houses.
‫So after we ingest this data from various sources, we collect it in a data lake.
‫We then perform something called data transformation that is converting data from one format to another,
‫usually into something we call a data warehouse.
‫A data warehouse is a place that stores accessible data that is useful for the business.
‫That is, although the lake collected so much water, so much data, that was kind of all over the place.
‫A data engineer looks at this data and forms or uses the parts of the data that are useful and puts
‫it in a data warehouse so that other parts of the business can use it again.
‫A good way to think about it is that a data lake is a pool of raw data.
‫The purpose of which we still don't know what to do with in a data warehouse is a location for structured,
‫filtered data that has been processed and has a specific purpose.
‫That is, we have data in here that is useful for the business.
‫So that means that data lakes are usually less organized and has less filtration of data than something
‫like a data warehouse.
‫Now, why would businesses want to do that?
‫Well, for one, it's a lot easier to analyze data when it's organized a well.
‫We also might have data in here that we don't really need.
‫We also save on storage space, right?
‫Because we don't have to store all of this data and just store structured data only that is only useful
‫to us.
‫We also save ourselves some money.
‫So to review a data engineer simply builds this pipeline of taking the data creation and data capture,
‫using data engineering practices to build this pipeline so that data can now be analyzed by data scientists
‫and data analysts.
‫But we still have a little bit more to go.
‫So I'll see you in the next video.