Delta Lake and Medallion Architecture for AI and ML

О курсе

This course expands your data engineering skills by focusing on Delta Lake and Medallion Architecture for building dependable, scalable, and analytics-ready data platforms. You will learn how modern pipelines maintain data quality, support version control, and prepare trusted datasets for AI/ML workloads. You will start with Delta Lake fundamentals, including ACID transactions, schema enforcement, schema evolution, transaction logs, and Time Travel. Through practical demonstrations, you will create Delta tables, manage updates and deletes, apply MERGE operations, and restore earlier versions of data when needed. You will then apply PySpark transformation techniques to clean, reshape, join, and validate datasets. You will also explore performance-focused practices such as OPTIMIZE, VACUUM, and Z-Ordering to make Delta tables more efficient for large-scale processing. Next, you will design Medallion Architecture pipelines using Bronze, Silver, and Gold layers. This helps convert raw data into clean, validated, and business-ready datasets for reporting, analytics, and machine learning. The course also introduces structured streaming, Change Data Feed, and Delta constraints for improving pipeline reliability. By the end of this course, you will be able to: - Use Delta Lake for reliable and versioned data management. - Transform and validate datasets using PySpark. - Optimise Delta tables for performance and storage efficiency. - Build Bronze, Silver, and Gold data layers. - Develop dependable batch and streaming pipelines for AI/ML use cases. Designed for data engineers, analytics engineers, software developers, and AI/ML professionals, this course prepares you to build modern data pipelines that are reliable, scalable, and ready for production workloads.

Навыки, которые вы освоите

Учитесь у экспертов

Edureka

Преподаватель курса