‫Let's talk about Apache Spark and Apache Flink.
‫You see, before Apache Spark, usually people use Hadoop, MapReduce to process everything.
‫Spark came along and actually improved the MapReduce and Hadoop by doing something called in-memory
‫processing, which essentially allowed Spark to run processing jobs much, much faster than MapReduce.
‫So Apache Spark became really, really popular, and it's probably the go to batch processing framework.
‫So if you want to process a lot of data, well, you can use Hadoop to store that data data in HDFS
‫and use Apache Spark to run ETL jobs like extract, transform, load to clean and transform that data.
‫Now, up until now, we had batch processing, which essentially means give me a chunk of data and I'll
‫process this data over a bit of time.
‫Usually you'd run a batch processing job at the end of the night and in the morning after a couple of
‫hours, the job is done.
‫But as of a few years ago, the idea of real time processing started to happen.
‫Things like spark streaming came out, which almost made real time processing possible.
‫That is, instead of batch processing where you waited until the end of the night to run some jobs on
‫amount of data.
‫Instead, every time data comes in, you process that data in this space.
‫Apache Flink really took on the charge.
‫It offers real time stream processing.
‫Now, although Spark is still popular, tools like Apache Flink are starting to become popular now.
‫This idea of streaming data and stream processing is becoming more and more popular.
‫It's still new, but there's some exciting things happening in this landscape.
‫So let's talk about that in the next video.