Julien Le Dem - The advent of the open data lake

Julien Le Dem is a Principal Engineer at Datadog, serves as an officer of the ASF and is a member of the LFAI&Data Technical Advisory Council. He co-created the Parquet, Arrow and OpenLineage open source projects and is involved in several others.

RSVP

The advent of the open data lake

Over the past decade, the big data ecosystem has matured and evolved from a melting pot of competing projects into a composable ecosystem organized around a few open source standards.

The components of databases, distributed or not, have been commoditized as individual parts that anyone can compose into use-case specific engines. Define your constraints and build a query engine that solves your problem.

It’s been incredible to see the adoption of key components like Parquet, Arrow, Iceberg, and DataFusion. They provide an interoperability layer that enables using data without creating silos and duplication.

In this talk he’ll discuss the impact of the cloud and the advent of the Open Data Lake breaking silos to form the foundation of this ecosystem. As compute and storage can be efficiently decoupled, a common storage layer enables a vibrant ecosystem of on-demand tools specialized to specific use cases that avoids vendor lock-in.