How the AST Matcher in dcg Uses Tree-Sitter to Detect Dangerous Operations in Embedded Scripts

The AST matcher in dcg uses Tree-Sitter via the ast-grep crate to parse embedded scripts into abstract syntax trees, applying pre-compiled destructive patterns to detect exec-sinks and filesystem sinks while refining matches based on literal payload analysis.

The Destructive Command Guard (dcg) protects AI-generated code by analyzing heredocs and inline scripts that may hide malicious commands. Its third-tier detection is performed by the AST matcher, which leverages Tree-Sitter to provide structural accuracy beyond simple regex matching. This article examines how the AST matcher in dcg uses Tree-Sitter to identify dangerous operations across multiple languages.

Architectural Overview of the AST Matcher

The AST matcher operates as a multi-stage pipeline defined in src/ast_matcher.rs. Each stage transforms raw script content into actionable security decisions.

Language Detection and Mapping

When src/heredoc.rs extracts a script from a heredoc, it identifies the programming language (e.g., Bash, Python, JavaScript) and passes the raw source to the matcher. The function script_language_to_ast_lang maps internal ScriptLanguage enums to ast-grep SupportLang variants. Unsupported languages gracefully fall back to regex-based detection.

// From src/ast_matcher.rs lines 80-92
fn script_language_to_ast_lang(lang: ScriptLanguage) -> Option<SupportLang> {
    match lang {
        ScriptLanguage::Python => Some(SupportLang::Python),
        ScriptLanguage::JavaScript => Some(SupportLang::JavaScript),
        ScriptLanguage::Bash => Some(SupportLang::Bash),
        // ... additional mappings
    }
}

Pattern Pre-compilation

During initialization, AstMatcher::new calls precompile_patterns to build a hash-map of destructive patterns for each supported language. This compiles Tree-Sitter queries at startup, eliminating per-request compilation overhead.

// From src/ast_matcher.rs lines 16-22
impl AstMatcher {
    pub fn new() -> Self {
        let patterns = precompile_patterns();
        Self {
            patterns,
            timeout: Duration::from_millis(20),
        }
    }
}

Time-Bounded AST Parsing

For each script analysis, run_ast_match_with_timeout spawns a worker thread that creates an AstGrep parser via AstGrep::new. A hard deadline (default 20 ms) protects the hook from hanging. The worker checks a shared AtomicBool after each pattern and node traversal via check_ast_timeout.

// From src/ast_matcher.rs lines 84-91 and 122-138
fn run_ast_match_with_timeout(&self, code: &str, lang: SupportLang) -> Result<Vec<PatternMatch>, MatchError> {
    // Spawn worker with timeout guard
    let result = thread::scope(|s| {
        s.spawn(|| {
            self.find_matches_ast(code, lang)
        }).join()
    });
}

Pattern Matching and Refinement

The core matching loop in find_matches_ast iterates over pre-compiled patterns, calling root.find_all(&compiled.pattern) for each. Every matched node yields its text, byte range, and line number. Language-specific refinement functions—refine_python_match, refine_javascript_match, refine_ruby_match—inspect literals inside matched nodes (e.g., payload strings) and upgrade severity from Medium to High or Critical based on content.

// From src/ast_matcher.rs lines 26-34
fn find_matches_ast(&self, code: &str, lang: SupportLang) -> Vec<PatternMatch> {
    let ast = AstGrep::new(code, lang);
    let root = ast.root();
    
    for pattern in &self.patterns {
        for node in root.find_all(&pattern.pattern) {
            // Extract match metadata...
        }
    }
}

Why Tree-Sitter Powers the Detection Engine

The AST matcher in dcg uses Tree-Sitter instead of regex for three critical reasons:

Structural Accuracy – Tree-Sitter parses source into a language-aware abstract syntax tree, enabling dcg to match specific call shapes (e.g., child_process.execSync(...)) while ignoring comments or innocuous string literals that would trigger false positives in regex systems.

Performance – Parsing a typical heredoc (< 1 MiB) costs approximately 2 ms, and pattern matching (< 1 ms) stays well under the 20 ms budget. This ensures the guard operates without perceptible latency in interactive AI coding sessions.

Safety – All errors (unsupported language, parse failure, timeout) trigger fail-open behavior: the command is allowed but logged, avoiding false negatives that could block legitimate workflows.

Detecting Dangerous Operations in Embedded Scripts

The matcher’s pattern set focuses on exec-sinks (functions that invoke a shell) and filesystem sinks (functions that delete files).

Exec-Sink Detection in JavaScript

Patterns for execSync or execFileSync are compiled via ast-grep. When a node matches, refine_javascript_match extracts the literal argument using JS_EXEC_SYNC_LITERAL and runs it through detect_shell_payload. If the payload contains destructive commands (rm -rf, git reset --hard), the severity upgrades to High or Critical.

Filesystem-Sink Detection in Ruby

Patterns targeting FileUtils.rm_rf match destructive file operations. The refinement checks the path literal; catastrophic paths (/, /etc, /home) raise severity to Critical and trigger immediate blocking.

Regex Fallback for Unsupported Languages

For languages lacking Tree-Sitter grammars (e.g., Perl), find_matches_perl provides a fast regex fallback. This ensures coverage without sacrificing safety when script_language_to_ast_lang returns None.

Code Examples

Create a matcher with default destructive patterns and analyze embedded Python:

// Create a matcher with the default destructive patterns
let matcher = AstMatcher::new();

// Analyse a heredoc body written in Python
let code = r#"
    python - <<'PY'
    import os
    os.system("rm -rf /")
    PY
"#;
let language = ScriptLanguage::Python;

// Find any blocking matches
if let Some(block) = matcher.has_blocking_match(code, language) {
    println!("Blocked! Rule: {}", block.rule_id);
    // → Blocked! Rule: heredoc.python.os_system.rm_rf
}

Configure custom timeouts and runtime patterns:

// Custom matcher with a shorter timeout (useful in tests)
let mut custom = AstMatcher::new()
    .with_timeout(Duration::from_millis(5));

// Add additional patterns at runtime
let mut extra = HashMap::new();
extra.insert(
    ScriptLanguage::JavaScript,
    vec![CompiledPattern::new(
        r"eval\(".into(),
        "heredoc.javascript.eval".into(),
        "JavaScript eval() call".into(),
        Severity::High,
        None,
    )],
);
let matcher = AstMatcher::with_patterns(extra);

Summary

• The AST matcher in dcg uses Tree-Sitter via ast-grep to parse embedded scripts in src/ast_matcher.rs, providing structural accuracy for detecting dangerous operations.
• script_language_to_ast_lang maps detected languages to Tree-Sitter grammars, while precompile_patterns optimizes startup performance.
• A 20 ms timeout guard in run_ast_match_with_timeout ensures the guard never hangs on malicious or malformed input.
• Language-specific refinement functions (refine_python_match, refine_javascript_match) analyze literal payloads to upgrade severity levels dynamically.
• Fail-open guarantees in src/evaluator.rs ensure that parse errors or timeouts result in logged allowances rather than false blocks.

Frequently Asked Questions

How does the AST matcher handle unsupported programming languages?

When script_language_to_ast_lang encounters a language without Tree-Sitter support (such as Perl), it returns None and triggers the regex fallback path. The find_matches_perl function applies pattern matching using regular expressions rather than AST analysis, ensuring the guard maintains coverage across all embedded scripts without crashing.

What happens if the Tree-Sitter parser takes too long to analyze a script?

The matcher enforces a hard timeout of 20 milliseconds by default through run_ast_match_with_timeout. If parsing or pattern matching exceeds this limit, the worker thread terminates and returns a timeout error. The calling code in src/evaluator.rs catches this error, allows the command to proceed, and logs the diagnostic, ensuring fail-open safety.

Can the AST matcher detect obfuscated commands in JavaScript heredocs?

Yes. The refinement system in refine_javascript_match extracts literal string arguments from matched nodes using JS_EXEC_SYNC_LITERAL and passes them to detect_shell_payload. This detects obfuscated destructive commands like rm -rf / even when they appear as concatenated strings or template literals inside child_process.execSync() calls.

Where does the AST matcher fit in the overall dcg evaluation pipeline?

The matcher operates as the third and final tier in src/evaluator.rs. After quick-reject filters, normalization, and safe-pattern matching, the evaluator calls the AST matcher for deep structural analysis of heredoc content. If has_blocking_match returns a PatternMatch with severity.blocks_by_default(), the command is denied and a JSON denial is formatted via src/output/denial.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}