data-engineering-zoomcamp

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼

35 articles 41.7k View on GitHub ↗
35 articles
Spark Coalesce vs Repartition: Understanding Data Shuffling Differences

Understand Spark coalesce vs repartition for data shuffling. Coalesce merges partitions locally avoiding shuffles, while repartition triggers a full shuffle for even distribution. Learn the key differences.

deep-dive
May 31, 2026
How Kestra Handles Task Dependencies and DAG Execution: A Complete Guide

Learn how Kestra manages task dependencies and DAG execution. Discover implicit and explicit dependency declarations for flexible workflow orchestration. Unlock sequential, parallel, and conditional patterns.

how-to-guide
May 31, 2026
How to Set Up CI/CD Pipelines for dbt with GitHub Actions

Learn to set up CI/CD pipelines for dbt with GitHub Actions. Automate dbt commands like build, test, and docs generation on every push for efficient data pipeline management.

how-to-guide
May 31, 2026
dbt Sources and Ref Functions: Understanding the Difference in Data Engineering

Understand dbt sources vs ref functions. Learn how `source()` references external tables and `ref()` points to dbt models. Optimize your data pipelines now.

deep-dive
May 31, 2026
How to Monitor and Alert on Kafka Consumer Lag Metrics in the Data Engineering Zoomcamp

Learn to monitor and alert on Kafka consumer lag using KafkaConsumer API, Prometheus for Spark, or Confluent Cloud integrations. Essential for data engineers.

how-to-guide
May 31, 2026
Trade-offs Between Batch and Streaming Processing Architectures in Data Engineering

Explore batch vs streaming processing trade-offs. Learn when to choose batch for cost-effective historical analysis or streaming for real-time insights. Optimize your data architecture.

deep-dive
May 31, 2026
How to Handle Late-Arriving Data and Watermarking in Streaming Pipelines

Master late-arriving data and watermarking in streaming pipelines. Learn how to safely close windows and emit aggregates with allowed lateness.

how-to-guide
May 31, 2026
Differences Between dbt Seeds, Macros, and Packages Explained

Understand the differences between dbt seeds, macros, and packages. Learn how seeds load CSVs, macros create reusable SQL, and packages extend your dbt project.

deep-dive
May 31, 2026
How to Set Up Kafka Connect for Streaming ETL to PostgreSQL

Set up Kafka Connect for streaming ETL to PostgreSQL easily. Persist Kafka JSON events to relational tables automatically using the JDBC Sink Connector. Learn how now.

how-to-guide
May 31, 2026
Common Pitfalls When Training BigQuery ML Models: 10 Critical Errors to Avoid

Avoid common BigQuery ML model training pitfalls like data leakage and NULL values. Learn defensive SQL patterns to improve performance and reduce costs. Explore solutions in DataTalksClub/data-engineering-zoomcamp.

best-practices
May 31, 2026
How to Secure Terraform State Files in GCP Cloud Storage: A Complete Guide

Learn to secure Terraform state files in GCP Cloud Storage. Follow our guide to harden your GCS bucket, enabling versioning, restricting IAM, and implementing lifecycle rules for ultimate protection.

how-to-guide
May 31, 2026
Docker Volumes vs Bind Mounts for PostgreSQL: Key Differences Explained

Understand Docker volumes vs bind mounts for PostgreSQL. Learn how Docker manages volumes internally and bind mounts give direct host access for your database.

deep-dive
May 31, 2026

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →