Data Lakes & Lakehouses

Understand how distributed file systems like HDFS and cloud object stores (S3, ADLS, GCS) work — block vs. object storage, eventual consistency, and how data is physically organized.

Learn why columnar formats like Parquet and ORC dominate analytical workloads — row groups, column chunks, encoding schemes, and how predicate pushdown works at the file level.

Master how partitioning by date, region, or other keys reduces scan volume, and understand the trade-offs between too many small files and too few large ones.

Compare the traditional data warehouse model (schema-on-write, structured) with the data lake approach (schema-on-read, multi-format) and understand where each excels.

Learn the multi-zone pattern — landing raw data, cleaning it into curated layers, and serving aggregated datasets for consumption — and why this layering prevents chaos.

Understand the key ingestion modes — scheduled batch loads, CDC streams, and micro-batch — and when to use each for landing data into a lake.

Learn how schemas drift over time in a lake, the problems this creates, and the mechanisms (schema registries, merge rules) that keep data compatible.

Understand how tools like AWS Glue Catalog, Hive Metastore, and Unity Catalog track table locations, schemas, and statistics so engines can discover and query lake data.

Identify the common failures that turn a data lake into an unusable swamp — missing metadata, no ownership, unbounded schema drift, and lack of quality checks.

Understand the core problem — ACID transactions, time travel, and efficient upserts on top of immutable object storage — and why plain Parquet directories aren't enough.

Learn Delta Lake's transaction log, optimistic concurrency, Z-ordering, VACUUM, and MERGE semantics. Understand its tight Spark integration and Databricks ecosystem.

Explore Iceberg's snapshot-based metadata tree, hidden partitioning, partition evolution, and multi-engine compatibility across Spark, Trino, Flink, and more.

Understand Hudi's copy-on-write vs. merge-on-read table types, its record-level indexing, incremental queries, and strengths for CDC-heavy workloads.

Evaluate the three major table formats side by side on concurrency, ecosystem support, partition evolution, and community momentum to make an informed choice.

Understand how the lakehouse combines the low-cost storage of a data lake with the reliability, performance, and governance features of a data warehouse.

Learn the medallion pattern for incrementally refining data quality — bronze for raw ingestion, silver for validated and conformed data, gold for business-level aggregates.

Survey the engines that query lakehouse tables — Spark, Trino/Presto, Dremio, StarRocks — and understand the trade-offs between throughput, latency, and cost.

Learn how to run interactive SQL over lakehouse tables, including BI tool integration, caching layers, and materialized views for sub-second dashboards.

Understand column-level and row-level security, role-based policies, and how catalogs like Unity Catalog or Apache Ranger enforce permissions across multiple engines.

Learn to define and enforce data quality rules — null checks, range validations, referential integrity — using tools like Great Expectations or Delta Live Tables expectations.

Trace how data flows from source to consumption, detect stale or broken pipelines, and integrate lineage tools like OpenLineage and DataHub into your lakehouse.

Understand why streaming ingestion creates many tiny files, how compaction (bin-packing) merges them, and how to schedule maintenance without disrupting readers.

Learn how min/max statistics, bloom filters, Z-ordering, and liquid clustering let engines skip irrelevant files and dramatically reduce I/O.

Explore local SSD caching, result caching, and acceleration layers (Alluxio, Delta Cache) that bridge the gap between object store latency and interactive query expectations.

Learn how Spark Structured Streaming, Flink, and Kafka Connect write directly into Delta/Iceberg tables with exactly-once semantics and low-latency availability.

Build CDC pipelines that replicate operational databases into lakehouse tables using Debezium, Airbyte, or native connectors, keeping the lake in near-real-time sync.

Understand how to use lakehouse tables as feature stores, run distributed training on lake data, and version ML datasets alongside model artifacts.