Engineering Data Pipelines

This is a reference guide for building production data pipelines, covering both batch orchestration (Airflow, Dagster, Prefect) and stream processing (Kafka Streams, Flink, Spark Streaming). It gives you checklists for the things that actually matter: idempotency patterns, incremental loads, watermark tuning for out-of-order events, state management in streaming jobs, and layered data quality checks with tools like Great Expectations and dbt tests. The bilingual formatting is a bit quirky, but the technical checklists are solid. Use this when you're setting up ETL workflows or real-time streaming and need a quick reference for best practices around exactly-once semantics, backpressure monitoring, or cross-DAG dependencies without digging through framework docs.

Install to Claude Code

npx -y skills add telagod/code-abyss --skill engineering-data-pipelines --agent claude-code