‫Let's finish up this high level overview of what a data engineer does by describing the three main tasks
‫that they do.
‫One is that they build what we call an ETL pipeline.
‫Or an extract transform load pipeline.
‫So unlike data ingestion, which simply means moving data from one place to another.
‫And ETL pipeline is the idea that a data engineer extracts data that has been generated by all of these
‫systems.
‫They extract that data, and then when they extract that data, they transform.
‫The data into a useful form that is into a form that can be loaded into a data warehouse.
‫So they extract the data.
‫They transform the data.
‫And they load it into something like a data warehouse so that the data can be used by the rest of the
‫company.
‫And they use programming languages like Python, Go Scala and Java to accomplish these ETL jobs or extract
‫transform load.
‫So that's the first task.
‫What's the second one?
‫The second one is building analysis tools.
‫You see for us to understand how our company systems work.
‫A data engineer needs to make sure that when any part of the system breaks, we are notified.
‫So a data engineer allows data scientists, data analysts, business intelligence people to use tools
‫to a analyze the data, but also to make sure that the system that they've put in place is running correctly.
‫Finally, their third main task is obviously to maintain the data warehouse and data lakes.
‫That is making sure that everything in there is accessible for other parts of the companies to use.
‫So now that we have a high level overview of what a data engineer does.
‫I want to talk about some of the tools that you have definitely heard of in the industry, things like
‫Hadoop, Kafka, Apache, Spark, all these terms that are really hard to understand.
‫I'm going to try and give you an overview of this landscape so that next time when somebody asks you,
‫Hey, should we use Hadoop?
‫Hey, what's Hadoop for?
‫Do we need Kafka?
‫You'll be able to answer that.
‫And after this course, you can go off on your own and actually learn to use these tools if they ever
‫come across in your job.
‫Now, this landscape is fast changing.
‫There's new tools popping up left, right and center.
‫So my advice to you is that what I'm going to show you over the next couple of videos you don't take
‫as the absolute must know for all data engineers.
‫Instead, know that they exist.
‫Go and read some of their documentation, some of the information online, but only learn or use them
‫once the need arises because they're constantly being updated.
‫And the world of data engineering is really fast paced right now.
‫But with that said, let's nerd out on data in the next couple of videos.