# data-engineering-zoomcamp | DataTalksClub | Knowledge Base | Instagit

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼

GitHub Stars: 41.7k

Repository: https://github.com/DataTalksClub/data-engineering-zoomcamp

---

## Articles

### [Spark Coalesce vs Repartition: Understanding Data Shuffling Differences](/DataTalksClub/data-engineering-zoomcamp/whats-the-difference-between-spark-coalesce-and-repartition-for-shuffling-data)

Understand Spark coalesce vs repartition for data shuffling. Coalesce merges partitions locally avoiding shuffles, while repartition triggers a full shuffle for even distribution. Learn the key differences.

- Tags: deep-dive
- Published: 2026-05-31

### [How Kestra Handles Task Dependencies and DAG Execution: A Complete Guide](/DataTalksClub/data-engineering-zoomcamp/how-does-kestra-handle-task-dependencies-and-dag-execution)

Learn how Kestra manages task dependencies and DAG execution. Discover implicit and explicit dependency declarations for flexible workflow orchestration. Unlock sequential, parallel, and conditional patterns.

- Tags: how-to-guide
- Published: 2026-05-31

### [How to Set Up CI/CD Pipelines for dbt with GitHub Actions](/DataTalksClub/data-engineering-zoomcamp/how-do-i-set-up-ci-cd-pipelines-for-dbt-with-github-actions)

Learn to set up CI/CD pipelines for dbt with GitHub Actions. Automate dbt commands like build, test, and docs generation on every push for efficient data pipeline management.

- Tags: how-to-guide
- Published: 2026-05-31

### [dbt Sources and Ref Functions: Understanding the Difference in Data Engineering](/DataTalksClub/data-engineering-zoomcamp/whats-the-difference-between-dbt-sources-and-ref-functions)

Understand dbt sources vs ref functions. Learn how `source()` references external tables and `ref()` points to dbt models. Optimize your data pipelines now.

- Tags: deep-dive
- Published: 2026-05-31

### [How to Monitor and Alert on Kafka Consumer Lag Metrics in the Data Engineering Zoomcamp](/DataTalksClub/data-engineering-zoomcamp/how-do-i-monitor-and-alert-on-kafka-consumer-lag-metrics)

Learn to monitor and alert on Kafka consumer lag using KafkaConsumer API, Prometheus for Spark, or Confluent Cloud integrations. Essential for data engineers.

- Tags: how-to-guide
- Published: 2026-05-31

### [Trade-offs Between Batch and Streaming Processing Architectures in Data Engineering](/DataTalksClub/data-engineering-zoomcamp/what-are-the-trade-offs-between-batch-and-streaming-processing-architectures)

Explore batch vs streaming processing trade-offs. Learn when to choose batch for cost-effective historical analysis or streaming for real-time insights. Optimize your data architecture.

- Tags: deep-dive
- Published: 2026-05-31

### [How to Handle Late-Arriving Data and Watermarking in Streaming Pipelines](/DataTalksClub/data-engineering-zoomcamp/how-do-i-handle-late-arriving-data-and-watermarking-in-streaming-pipelines)

Master late-arriving data and watermarking in streaming pipelines. Learn how to safely close windows and emit aggregates with allowed lateness.

- Tags: how-to-guide
- Published: 2026-05-31

### [Differences Between dbt Seeds, Macros, and Packages Explained](/DataTalksClub/data-engineering-zoomcamp/what-are-the-differences-between-dbt-seeds-macros-and-packages)

Understand the differences between dbt seeds, macros, and packages. Learn how seeds load CSVs, macros create reusable SQL, and packages extend your dbt project.

- Tags: deep-dive
- Published: 2026-05-31

### [How to Set Up Kafka Connect for Streaming ETL to PostgreSQL](/DataTalksClub/data-engineering-zoomcamp/how-do-i-set-up-kafka-connect-for-streaming-etl-to-postgresql)

Set up Kafka Connect for streaming ETL to PostgreSQL easily. Persist Kafka JSON events to relational tables automatically using the JDBC Sink Connector. Learn how now.

- Tags: how-to-guide
- Published: 2026-05-31

### [Common Pitfalls When Training BigQuery ML Models: 10 Critical Errors to Avoid](/DataTalksClub/data-engineering-zoomcamp/what-are-common-pitfalls-when-training-bigquery-ml-models)

Avoid common BigQuery ML model training pitfalls like data leakage and NULL values. Learn defensive SQL patterns to improve performance and reduce costs. Explore solutions in DataTalksClub/data-engineering-zoomcamp.

- Tags: best-practices
- Published: 2026-05-31

### [How to Secure Terraform State Files in GCP Cloud Storage: A Complete Guide](/DataTalksClub/data-engineering-zoomcamp/how-do-i-secure-terraform-state-files-in-gcp-cloud-storage)

Learn to secure Terraform state files in GCP Cloud Storage. Follow our guide to harden your GCS bucket, enabling versioning, restricting IAM, and implementing lifecycle rules for ultimate protection.

- Tags: how-to-guide
- Published: 2026-05-31

### [Docker Volumes vs Bind Mounts for PostgreSQL: Key Differences Explained](/DataTalksClub/data-engineering-zoomcamp/what-are-the-differences-between-docker-volumes-and-bind-mounts-for-postgresql)

Understand Docker volumes vs bind mounts for PostgreSQL. Learn how Docker manages volumes internally and bind mounts give direct host access for your database.

- Tags: deep-dive
- Published: 2026-05-31

### [Spark DataFrame Optimization and Caching: 7 Best Practices from the Data Engineering Zoomcamp](/DataTalksClub/data-engineering-zoomcamp/what-are-best-practices-for-spark-dataframe-optimization-and-caching)

Master Spark DataFrame optimization with 7 best practices. Learn to minimize shuffles, cache wisely, and tune configurations for peak performance from DataTalksClub. Accelerate your data pipelines.

- Tags: best-practices
- Published: 2026-05-31

### [How Kafka Handles Schema Evolution with Avro Schemas: A Complete Guide](/DataTalksClub/data-engineering-zoomcamp/how-does-kafka-handle-schema-evolution-with-avro-schemas)

Efficiently manage schema evolution in Kafka using Avro and the Confluent Schema Registry. Learn how versioned schemas ensure backward and forward compatibility for seamless data integration.

- Tags: how-to-guide
- Published: 2026-05-31

### [Trade‑offs Between Clustering and Partitioning in BigQuery](/DataTalksClub/data-engineering-zoomcamp/what-are-the-trade-offs-between-clustering-and-partitioning-in-bigquery)

Understand BigQuery partitioning vs clustering. Use partitioning for time-based pruning and clustering for block-level filtering. Learn the optimal combination for analytics.

- Tags: deep-dive
- Published: 2026-05-31

### [How to Configure dbt with BigQuery for Production Deployments](/DataTalksClub/data-engineering-zoomcamp/how-do-i-configure-dbt-with-bigquery-for-production-deployments)

Learn to configure dbt with BigQuery for production. Set up service-account authentication and reference your profile for smooth deployments.

- Tags: how-to-guide
- Published: 2026-05-31

### [How BigQuery Partitioning Improves Query Performance: Partition Pruning Explained](/DataTalksClub/data-engineering-zoomcamp/how-does-bigquery-partitioning-improve-query-performance)

Learn how BigQuery partitioning improves query performance through partition pruning. Scan less data, reduce costs, and speed up queries by understanding this essential optimization technique.

- Tags: deep-dive
- Published: 2026-05-31

### [How to Handle Late-Arriving Data and Out-of-Order Events in Streaming Pipelines with Kafka](/DataTalksClub/data-engineering-zoomcamp/handle-late-arriving-data-streaming-pipelines-kafka)

Master late-arriving data and out-of-order events in Kafka streaming pipelines. Learn event-time processing with watermarks and upsert sinks for reliable data pipelines.

- Tags: how-to-guide
- Published: 2026-05-30

### [How to Set Up Alerts and Monitoring for Data Pipeline Failures in Kestra: A Complete Guide](/DataTalksClub/data-engineering-zoomcamp/alerts-monitoring-data-pipeline-failures-kestra)

Instantly detect Kestra data pipeline failures with built-in hooks and Prometheus metrics. Send alerts to Slack, email, or webhooks and ensure data reliability.

- Tags: how-to-guide
- Published: 2026-05-30

### [How to Implement Near Real-Time Data Ingestion from APIs Using DLT with CDC](/DataTalksClub/data-engineering-zoomcamp/real-time-data-ingestion-apis-dlt-cdc)

Learn to implement near real-time data ingestion from APIs using DLT with CDC. Capture API changes in minutes with `rest_api_source`, `incremental` modes, and `continuous` refresh for exact data.

- Tags: how-to-guide
- Published: 2026-05-30

### [How to Backfill Historical Data in BigQuery Cost-Effectively: A Production Guide](/DataTalksClub/data-engineering-zoomcamp/backfill-historical-data-bigquery-cost-effectively)

Backfill historical data in BigQuery efficiently using external tables, partitioning, and idempotent MERGE statements. Orchestrate with Kestra for cost savings.

- Tags: how-to-guide
- Published: 2026-05-30

### [How to Debug Failed Kestra Flow Executions in Data Engineering Zoomcamp](/DataTalksClub/data-engineering-zoomcamp/debug-failed-kestra-flow-executions)

Debug failed Kestra flow executions in your Data Engineering Zoomcamp project Monitor task logs use the REST API check Docker connectivity and implement onError handlers for robust data pipelines.

- Tags: how-to-guide
- Published: 2026-05-30

### [How to Manage Terraform State in Collaborative Data Engineering: Best Practices from the Zoomcamp](/DataTalksClub/data-engineering-zoomcamp/terraform-state-management-collaborative-data-engineering)

Master Terraform state management for collaborative data engineering. Discover best practices for remote backends, environment isolation, and CI/CD integration to streamline team workflows.

- Tags: best-practices
- Published: 2026-05-30

### [How Kestra's Concurrency Control Prevents Duplicate Data Processing](/DataTalksClub/data-engineering-zoomcamp/kestra-concurrency-control-prevent-duplicate-data-processing)

Discover how Kestra's concurrency control stops duplicate data processing. Learn how it enforces limits to ensure precise pipeline execution and data integrity.

- Tags: how-to-guide
- Published: 2026-05-30

### [Designing Kafka Topics for Streaming Taxi Data: A Production Guide](/DataTalksClub/data-engineering-zoomcamp/designing-kafka-topics-streaming-taxi-data)

Design Kafka topics for streaming taxi data. Learn production best practices for partitioning, schema enforcement, and security to build scalable, fault-tolerant analytics.

- Tags: best-practices
- Published: 2026-05-30

### [How to Implement Incremental Models in dbt Using incremental_strategy](/DataTalksClub/data-engineering-zoomcamp/implement-incremental-models-dbt-incremental_strategy)

Implement dbt incremental models efficiently using incremental_strategy merge. Process only new or changed rows with the is_incremental macro for optimized data pipelines.

- Tags: how-to-guide
- Published: 2026-05-30

### [Spark Broadcast Joins vs Sort-Merge Joins: Execution Strategies and Optimization Guide](/DataTalksClub/data-engineering-zoomcamp/spark-broadcast-joins-vs-sort-merge-joins)

Understand Spark broadcast joins vs sort-merge joins Explore execution strategies and optimization guide for efficient data processing. Learn when to use each join type.

- Tags: deep-dive
- Published: 2026-05-30

### [How to Set Up CI/CD Pipelines for dbt Projects: A Complete Guide](/DataTalksClub/data-engineering-zoomcamp/setup-ci-cd-pipelines-dbt-projects)

Automate dbt projects with CI/CD pipelines. Learn to set up GitHub Actions for dependency installation, model building, testing, and documentation generation on every push or pull request.

- Tags: how-to-guide
- Published: 2026-05-30

### [Common Causes of Out-of-Memory Errors in Apache Spark and How to Prevent Them](/DataTalksClub/data-engineering-zoomcamp/spark-out-of-memory-errors-prevention)

Fix common out-of-memory errors in Apache Spark. Learn to prevent JVM heap exhaustion with optimized memory allocation, avoiding collect and tuning caching and serialization.

- Tags: how-to-guide
- Published: 2026-05-30

### [How Kafka Handles Schema Evolution with Avro Schemas: A Complete Implementation Guide](/DataTalksClub/data-engineering-zoomcamp/kafka-schema-evolution-avro-schemas)

Learn how Kafka handles schema evolution with Avro and Schema Registry. Understand schema IDs, compatibility rules, and seamless data processing for producers and consumers.

- Tags: how-to-guide
- Published: 2026-05-30

### [How to Implement Data Quality Checks in dbt with Custom Tests and Assertions](/DataTalksClub/data-engineering-zoomcamp/implement-data-quality-checks-dbt-custom-tests)

Learn to implement robust data quality checks in dbt using custom Jinja macros and assertions. Ensure data accuracy and reliability with effective testing strategies.

- Tags: how-to-guide
- Published: 2026-05-30

### [Security Best Practices for Storing GCP Service Account Credentials in Kestra](/DataTalksClub/data-engineering-zoomcamp/security-best-practices-gcp-service-account-credentials-kestra)

Learn security best practices for storing GCP service account credentials in Kestra. Securely manage keys in Kestra's Secret Store to prevent leakage and simplify rotation.

- Tags: best-practices
- Published: 2026-05-30

### [Difference Between ETL and ELT Patterns in Data Engineering: A Practical Guide](/DataTalksClub/data-engineering-zoomcamp/etl-vs-elt-patterns-data-engineering)

Understand the difference between ETL and ELT data engineering patterns. Learn how ETL transforms data before loading, and ELT transforms data within the warehouse for efficient data pipelines.

- Tags: tutorial
- Published: 2026-05-30

### [Best Practices for Partitioning and Clustering in BigQuery for Optimal Query Performance](/DataTalksClub/data-engineering-zoomcamp/best-practices-for-partitioning-and-clustering-in-bigquery)

Optimize BigQuery query performance with partitioning and clustering best practices. Learn how to minimize bytes scanned and reduce costs by structuring your data effectively.

- Tags: best-practices
- Published: 2026-05-30

### [Kestra Task Dependencies and Execution Ordering in Complex ETL Pipelines](/DataTalksClub/data-engineering-zoomcamp/how-does-kestra-handle-task-dependencies-and-execution-ordering-in-complex-etl-pipelines)

Learn how Kestra expertly manages task dependencies and execution ordering in complex ETL pipelines. Discover DAGs, explicit dependencies, and runtime data flow control for efficient workflows.

- Tags: how-to-guide
- Published: 2026-05-30