‫Let's talk about Hadoop.
‫So as companies started to generate more and more data, our databases that we had, like MySQL, became
‫too inefficient or unable to hold the large amounts of data.
‫And Hadoop became a cornerstone of data engineering.
‫You see, Hadoop is an open source distributed processing framework.
‫It allows us to do data processing and storage for big data.
‫And it was actually developed by Yahoo!
‫Back in the day and then later donated to the Apache Software Foundation.
‫Hadoop was all the rage when the big data craze started.
‫Hadoop was a solution that allowed all these companies with petabytes of data to store this information
‫in Hadoop.
‫Now, Hadoop was essentially a data lake, right?
‫It's a data lake solution.
‫It allows us to just store all this data for us.
‫But the popularity of Hadoop was because of two big things.
‫One was HD FS.
‫That is the Hadoop distributed file system.
‫Hadoop was able to store so much data because of this HDFS, a file system just like you have on your
‫computer that allowed it to store files on multiple computers.
‫That is, we can use Hadoop on multiple computers to store as much data as we want.
‫It was scalable.
‫We could scale out.
‫So data is stored across different physical computers.
‫Then was MapReduce, because once we store data, we need to perform some jobs, some processing on
‫that data.
‫Right.
‫And MapReduce in Hadoop allowed us to perform jobs against this data that we had in a data lake using
‫languages like Java or Python.
‫You can use MapReduce in Hadoop to do what we call batch processing.
‫That is, every night we can run some sort of a job to clean our data, to process our data and make
‫it useful for the company.
‫Now, MapReduce isn't used as often anymore as we'll see later on in the videos because we have something
‫called Apache Spark that's actually a little bit faster.
‫But we'll get to that.
‫The idea here is that we finally had a tool with Hadoop that allows us to store a lot of data across
‫multiple machines using the HDFS and Ron batch jobs using MapReduce on this large sets of data and a
‫lot of other tools started popping up around Hadoop tools such as Hive.
‫Love their logo, by the way.
‫Check that out.
‫Anyway.
‫Hive makes your Hadoop cluster.
‫Feel like it's a relational database.
‫Even though it isn't.
‫It allows you to use Hive and write SQL queries against your HDFS file system.
‫So in a sense, you kind of had this data lake of Hadoop and also a little bit of a data warehouse type
‫thing with Hive by allowing SQL commands on your data that hopefully was processed using MapReduce.
‫And you may not have heard of Hadoop before because it's used by data engineers.
‫Remember, data engineers usually don't work on systems that are customer facing.
‫That is, they don't build databases that you would use with your app or with a web app.
‫Instead, they build the systems behind the scenes that ingests, collects, runs, ETL, jobs, that
‫is extract, transform, load on data.
‫So Hadoop, is that behind the scenes storage layer that stores large amount of data running batch processing
‫against this data so that we can have some useful information.
‫And Hadoop is still very popular and very capable when it comes to large data, petabytes of data.
‫Let's learn a few more tools in the next video.