‫Okay.
‫So what does a data engineer actually do?
‫Let's start off with three big words that we need to understand before we really understand what a data
‫engineer does.
‫Yes, I know.
‫I keep circling around data engineer and I keep getting you excited, but I promise we're getting really,
‫really close.
‫We just need to discuss these three words that you may or may not have heard of.
‫One is data mining, and data mining simply means pre-processing and extracting some knowledge from
‫the data.
‫So we use some sort of data to extract knowledge from.
‫That's what data mining is.
‫Big data?
‫Well, it means that we have a lot of data, a lot of variables, so much data, in fact, that you
‫can't run on your laptop.
‫There's just too much data for your laptop or computer to hold.
‫Usually when we talk about big data, it's data that's so big that you need to have it running on cloud
‫computing or multiple computers such as a AWS Azure and Google Cloud because they have a lot of computers
‫and a lot of storage to store this data.
‫So that's why big data is big data is data that you usually can't have just on a one computer.
‫This usually happens because, well, data sets gets so big.
‫We have petabytes and petabytes of data now.
‫So just having data in a database like MySQL or post grass or any type of database becomes really difficult
‫when that database is just one single machine.
‫Now, new technologies were invented to solve this problem of big data like Hadoop and NoSQL, which
‫we'll talk about.
‫Now a data pipeline is essentially a pipeline that a data engineer builds to essentially use the fact
‫that we have this big amount of data, we need to extract information from this data using data mining.
‫So we need to bring or build a pipeline that allows us to flow from that unknown large amount of data
‫to a pipeline that extracts data to a more useful form.
‫So a data engineer essentially does this creates a data pipeline where all the information that different
‫devices like Iot devices, mobile applications, web apps, cameras, cars and pretty much anything
‫that collects data and stores information or logs data into servers or to the cloud.
‫A data engineer essentially accumulates all this information into nicely packed databases and storage
‫engines so that different parts of the company can create visualizations, they can monitor the performance
‫of their product, they can get business insights and make business decision from this data and even
‫use this data on their apps.
‫For example, for user profiles before a data scientist or a machine learning expert or even a business
‫intelligence or data analyst gets hired for a big company.
‫The thing that they need to do before all of that can be done is to hire a data engineer.
‫They build the pipeline that allows us to work as data scientists.
‫Remember this diagram and how we've been focusing on this part.
‫Well, a data engineer allows us to do this data collection part.
‫They bring in all this information, organize it in a way for us to do our data modeling.
‫They really help us with this data collection part that usually as a machine learning engineer or data
‫scientist, you don't have to concern yourself too much with.
‫But you know what?
‫My favorite demonstration of what a data engineer is is in the next video.
‫I hope you're excited because things are going to start to make more and more sense.
‫I'll see you on that one.
‫Bye bye.