Databricks Pipelines

This is your go-to for building data pipelines on Databricks using Lakeflow (formerly Delta Live Tables). It handles the critical decision of streaming table versus materialized view, which trips up most people. The decision tree alone is worth having open while you work. Covers Auto Loader for ingestion, Auto CDC for change data capture, Append Flows for multi-source patterns, and expectations for data quality. One thing that stands out: it explicitly calls out common traps like trying to aggregate over streaming sources (use an MV with batch read instead) and misusing unions when you need Append Flows. Requires databricks CLI and assumes you're already familiar with core Databricks operations.