There's a reason you're taking this course. Either you've seen that there's an explosion of "Data Engineers" in the data industry, you've seen a lot of the technical excitement or your organisation has decided to adopt a new Spark-based platform... But why is this trend happening? Why are people moving to lake-based platforms?

To understand this, we need to dive into the world of big data and the cloud, we need to evaluate what's wrong with the status quo, and what we want in an ideal new platform.

We then look at Spark, how it's grown and evolved, and how the idea of lakes has evolved as technology has matured. Finally, we'll set the scene for the Lakehouse and where this thinking has come from.