How ProjectBuilder Builds and Validates a DAT Project: A Deep Dive into the junjiem/dat SDK

The ProjectBuilder class orchestrates a seven-step incremental pipeline that loads DAT project configuration, computes content fingerprints to detect file changes, and executes comprehensive semantic validation via PreBuildValidator before persisting artifacts to the content store.

The ProjectBuilder located at ai.dat.boot.ProjectBuilder serves as the primary entry point for transforming DAT project source files into executable artifacts within the junjiem/dat repository. According to the source code, this builder implements an intelligent incremental build system that only validates and rebuilds when source files change, optimizing development workflows for data analytics projects while ensuring semantic correctness through rigorous pre-build checks.

The Seven-Step Build Pipeline in ProjectBuilder

The ProjectBuilder.build() method in dat-sdk/src/main/java/ai/dat/boot/ProjectBuilder.java orchestrates the entire construction process through seven distinct phases that balance performance with thoroughness:

1. Load project configuration: Calls ProjectUtil.loadProject to parse dat_project.yaml into a DatProject model (lines 45-50).
2. Compute fingerprint: Generates a content hash of all source files using ProjectUtil.contentStoreFingerprint to detect changes (lines 51-53).
3. Load previous state: Retrieves the last build's SchemaFileState list via BuildStateManager.loadBuildState (lines 53-55).
4. Detect file changes: Compares current hashes with stored state using FileChangeAnalyzer.analyzeChanges (lines 55-57).
5. Validate semantics: If changes exist, runs PreBuildValidator.validate to enforce configuration sanity and SQL correctness (lines 58-61).
6. Update content store: Persists new file hashes and compiled artifacts via ContentStoreManager.updateStore (lines 62-64).
7. Log completion: Emits build completion status (lines 65-66).

For forced rebuilds, ProjectBuilder.forceRebuild() first invokes cleanState() to clear stored fingerprints before executing the standard pipeline, ensuring a clean slate regardless of previous build history.

How PreBuildValidator Validates DAT Project Semantics

The PreBuildValidator class in dat-sdk/src/main/java/ai/dat/boot/PreBuildValidator.java enforces semantic correctness through multiple validation layers that catch errors before they reach the execution phase:

• Configuration validation: FactoryUtil.validateFactoryOptions verifies required options exist and optional parameters have correct types (lines 54-59).
• Template rendering: JinjaTemplateUtil.render processes Jinja templating in model definitions using provided variables (lines 64-70).
• Database adapter creation: ProjectUtil.createDatabaseAdapter instantiates concrete adapters (MySQL, PostgreSQL, DuckDB) based on project configuration (lines 76-78).
• Raw SQL validation: validateModelSql executes SELECT 1 FROM (<model>) to verify SQL syntax parses correctly on the target database (lines 17-26).
• Semantic model validation: validateSemanticModelSql uses SemanticModelUtil.semanticModelSql to translate models into runnable SQL, then executes WITH ... SELECT 1 FROM ... guard queries (lines 54-72).
• Dimension enum validation: validateDimensionEnumValues checks that declared enum values exist in underlying data and that distinct counts remain below safe thresholds (lines 200-236).
• Data type verification: validateDataTypes retrieves column metadata and ensures manually declared types match actual database column types (lines 97-108).
• Auto-completion: autoCompleteDataTypes fetches ANSI SQL types from the database for elements lacking declared dataType and writes them back into the semantic model (lines 124-136).

All validation failures throw ValidationException with human-readable error lists, aborting the build before ContentStoreManager persists any artifacts.

Code Example: Building a DAT Project with ProjectBuilder

The following Java example demonstrates the public API for building a DAT project:

import ai.dat.boot.ProjectBuilder;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Map;

public class BuildDemo {
    public static void main(String[] args) throws Exception {
        // 1️⃣ Project root (the directory that contains `dat_project.yaml`)
        Path projectPath = Paths.get("./my-dat-project").toAbsolutePath();

        // 2️⃣ Create the builder
        ProjectBuilder builder = new ProjectBuilder(projectPath);

        // 3️⃣ Optional runtime variables – they are injected into Jinja templates
        Map<String, Object> variables = Map.of(
                "tenantId", 42,
                "userId",   1001
        );

        // 4️⃣ Run an incremental build (validation runs only if files changed)
        builder.build(variables);

        // 5️⃣ Force a full rebuild ignoring previous state
        // builder.forceRebuild(variables);
    }
}

This implementation mirrors the public API documented in ProjectBuilder and shows how runtime variables flow into the validation and rendering phase.

Summary

• ProjectBuilder orchestrates a seven-step pipeline that loads configuration, detects file changes via content hashing, and conditionally validates before persisting artifacts.
• PreBuildValidator enforces semantic correctness through SQL syntax checks, database adapter validation, Jinja template rendering, and optional data type verification.
• The build process is incremental by default, only running expensive validation when FileChangeAnalyzer detects modifications, with forceRebuild() available for clean builds.
• All validation errors throw ValidationException before any artifacts are written, ensuring the content store never contains invalid state.

Frequently Asked Questions

What triggers a validation in ProjectBuilder?

Validation runs only when FileChangeAnalyzer.analyzeChanges detects added, modified, or deleted files by comparing current content fingerprints against the previous build state stored by BuildStateManager. If no changes exist, the build skips validation and completes immediately.

How does ProjectBuilder detect file changes?

The builder generates a cryptographic hash of all source files using ProjectUtil.contentStoreFingerprint during the load phase. It then retrieves the previous build's SchemaFileState list via BuildStateManager.loadBuildState and delegates comparison logic to FileChangeAnalyzer.analyzeChanges to identify specific changes.

What happens if validation fails during the build?

Any validation failure in PreBuildValidator.validate throws a ValidationException containing a human-readable list of errors. The build aborts immediately before ContentStoreManager.updateStore persists any artifacts, ensuring the project state remains consistent and invalid configurations never reach the content store.

Can I force a full rebuild without incremental checks?

Yes. ProjectBuilder.forceRebuild() removes the stored build state by calling cleanState() before executing the standard build pipeline. This forces the builder to treat all files as changed, running full validation and regeneration regardless of previous fingerprints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}