‫Welcome back.
‫So we just went through this whole flow of how data is used by a company.
‫But there's a few more things I want to talk about, because right now we just talked about water and
‫how does this really apply?
‫What kind of tools do data engineers actually use?
‫Well, you may have heard of something like.
‫Kafka and we'll get to Kafka later on.
‫But something like collecting these data streams and putting them into a data lake.
‫Might be something that Kafka or Apache Kafka might do for data lakes, for example.
‫These are simply programs that companies can use.
‫For example, you may have heard of Hadoop or Amazon S3 or the Azure Data Lake.
‫These are programs that have been built by engineers to hold large amounts of data, like a data leak.
‫And then for our data warehouse, we have tools like Google, BigQuery, Amazon, Redshift, even Amazon,
‫Athena.
‫These are data warehouses that allow engineers to make queries or analyze this structured data.
‫Now I know what you're thinking.
‫Who does what in this whole system?
‫We've learned that the data engineer creates this entire system for us.
‫They use different tools and programs to ingest data and then put it into a data lake or a data warehouse.
‫So as a data scientist and a machine learning expert.
‫Which data do you use?
‫Pause the video and think about it.
‫Ready for the answer.
‫Well, most of the time, you'd be working with a data lake.
‫Because remember, if you're doing machine learning, the more data you have, the better, right?
‫And with machine learning you can use structured or unstructured data.
‫So you can go into a data lake and actually grab a bunch of data to use for your models, whether they're
‫in CSV forms or any other forms.
‫And usually data warehouses are used by babies or business intelligent people or business analysts or
‫data analysts to make visualisation or analyze data.
‫Because the data warehouse has usually more structured data that has been cleaned out than it was when
‫it was a data leak.
‫It's a lot easier to understand and use the data for information.
‫Now, as a data scientist, you can use the data from a data warehouse.
‫And this isn't just a rule, it's usually you use whatever data is usable to you.
‫But this is a good way to think about things that a data scientist uses as much data that they can,
‫as much useful data as they can.
‫While somebody like a business intelligence person or a data analyst has already the data cleaned process
‫by a data engineer and use something like a data warehouse to analyze data.
‫And Google's big query is exactly that.
‫It allows somebody with not too much engineering experience or programming experience to analyze this
‫data in a data warehouse.
‫Another way to think about it is like this usually a software engineer, a software developer, app
‫developer, mobile developer, they build programs and apps that users and customers use and that releases
‫data.
‫Then a data engineer would build this piping and pipeline for us to ingest data and store it in different
‫services like Hadoop, like Google, BigQuery, so that that data can be accessed by the rest of the
‫business.
‫Next, we have data scientists that use the data lake, as well as the data scientists to extract information
‫and deliver some sort of business value.
‫Finally, we have data analysts or business intelligence to use something like a data warehouse or structured
‫data to, again, derive business value.
‫And I hope that clears the picture a little bit when you hear all these titles out there in the industry.
‫Now, the industry is fast evolving and there's definitely some overlap.
‫And sometimes what one job description might say might be different than the other.
‫But these are general, simplified rules that you can use to understand how each role plays into the
‫part of a company.