Engineering Data Pipelines

telagod/code-abyss

This is a reference guide for building production data pipelines, covering both batch orchestration (Airflow, Dagster, Prefect) and stream processing (Kafka Streams, Flink, Spark Streaming). It gives you checklists for the things that actually matter: idempotency patterns, incremental loads, watermark tuning for out-of-order events, state management in streaming jobs, and layered data quality checks with tools like Great Expectations and dbt tests. The bilingual formatting is a bit quirky, but the technical checklists are solid. Use this when you're setting up ETL workflows or real-time streaming and need a quick reference for best practices around exactly-once semantics, backpressure monitoring, or cross-DAG dependencies without digging through framework docs.

GitHub stars

224

last updated

Jun 10, 2026

编排：Airflow(调度) | Dagster(资产) | Prefect(现代流)
流处理：Kafka Streams(嵌入式) | Flink(集群) | Spark Streaming
质量：Great Expectations | dbt tests | Soda Core

Sponsors

SnugThe clipboard manager you and your agent can use, safelyAgent CaffeineClose your lid, come back to finished workWaxBlock anything with supervision, without erasing your iPhoneOpen StepsSponsoredFree and open-source skills that make AI coding agents easier to understand, verify, and control.