Lisa N. Cao - Designing Data Infrastructure in the Age of Generative AI

Lisa is an engineering, product, and advocacy expert in open source data infrastructure and DataOps fields.

At Databricks, her role oversees the open source involvement and developer relations of projects including MLflow, Apache Spark™, Delta Lake, Apache Iceberg™, and Unity Catalog OSS.

She also serves on the LF AI & Data Governing Board, formerly led the Open Platform for Enterprise AI's (OPEA) Developer Experience Working Group, and leads the Continuous Delivery Foundation's (CDF) DataOps Initiative.

RSVP

Designing Data Infrastructure in the Age of Generative AI

Developing powerful Al tooling has been our theme of the year, with agents and foundational models picking up steam across the board.

Therein still lies the question though: how do we serve data for agents to work effectively? What sort of interfaces and service mesh infrastructures will be required? What about at enterprise scale? What even is context?

In this talk we discuss the current big data landscape, challenges to data platforming for Al, and the shifting importance of open table formats, catalogs, and embedded systems as means for effective, governed Al-development. In this talk we use the open source technologies such as Apache Spark™, Unity Catalog OSS, and Apache Iceberg™ as key components of such reference architecture.