How AST-Based Pattern Matching Works for Embedded Scripts in Heredocs

AST-based pattern matching in destructive_command_guard parses extracted script content into abstract syntax trees using ast-grep to distinguish actual code from string literals and comments, enabling high-confidence detection of destructive commands hidden in shell heredocs.

The destructive_command_guard (dcg) repository implements a sophisticated three-tier pipeline to isolate dangerous commands embedded in shell heredocs or inline-script flags (e.g., python -c …). The third and final tier leverages AST-based pattern matching to perform structural analysis of extracted scripts, eliminating false positives that regular expressions alone cannot catch while maintaining sub-20-millisecond performance through pre-compiled patterns and strict timeout budgets.

The Three-Tier Detection Pipeline

The detection architecture splits analysis into distinct stages, each defined in src/heredoc.rs. This design ensures zero-allocation fast-paths for safe commands while providing deep inspection when needed.

Tier 1: Trigger Scanning

The check_triggers function uses a RegexSet to perform an ultra-fast scan for heredoc delimiters or inline-script indicators. This stage allocates no memory on the fast-path and immediately returns control for commands without embedded scripts.

Tier 2: Content Extraction

When triggers fire, extract_content performs bounded extraction of the heredoc body or inline-script content. This enforces limits on total size, line count, and the number of heredocs per command, plus a strict timeout to prevent denial-of-service via malicious input.

Tier 3: AST-Based Matching

The extracted script content passes to the AstMatcher in src/ast_matcher.rs. This structural analyzer gives the highest-confidence verdict by parsing the code into an abstract syntax tree (AST) and matching against language-specific destructive patterns.

Inside the AST Matcher Implementation

The core matching logic resides in the AstMatcher struct, which orchestrates parsing, pattern matching, and result refinement.

Creating the Matcher

The AstMatcher struct maintains pre-compiled, language-specific patterns and a hard-coded timeout budget:

pub struct AstMatcher {
    /// Patterns organised by language.
    patterns: HashMap<ScriptLanguage, Vec<PrecompiledPattern>>,
    /// Timeout for matching operations.
    timeout: Duration,
}

During construction (AstMatcher::new), the system loads default destructive patterns from src/config.rs, calls precompile_perl_patterns() to handle Perl safely, and stores the compiled results via precompile_patterns(default_patterns()). The production timeout is set to 20 ms (AST_TIMEOUT_MS) to ensure interactive shell performance.

The Matching Process

The public API find_matches processes extracted content through four guarded steps:

1. Language support validation – Maps ScriptLanguage to ast_grep_language::SupportLang, returning UnsupportedLanguage for unrecognized languages.

2. Pattern retrieval – Fetches pre-compiled patterns from the HashMap; empty pattern lists return early with Ok(Vec::new()).

3. Resource guards – Aborts if input exceeds MAX_AST_INPUT_BYTES or if the timeout budget expires before parsing begins.

4. Worker thread execution – Spawns a dedicated thread via run_ast_match_with_timeout to run the actual AST parsing. A shared AtomicBool allows the parent to cancel the worker immediately upon timeout, preventing runaway parsing from blocking the shell.

Inside the worker, the code constructs the syntax tree:

let ast = AstGrep::new(code, ast_lang);
let root = ast.root();

The matcher walks this tree using root.find_all(&compiled.pattern) for each pre-compiled rule, recording matches with their source range, line number, and a truncated preview.

Refinement and Context Analysis

Raw AST matches undergo language-specific refinement to eliminate false positives. Functions like refine_python_match and refine_javascript_match (starting around line 640 in src/ast_matcher.rs) analyze the match context to distinguish between actual destructive calls and identical strings inside comments or literals. This refinement ensures that rm -rf / inside a Python docstring does not trigger a block, while os.system("rm -rf /") in executable code does.

Safety Mechanisms and Fallbacks

If AST parsing fails due to unsupported language syntax, parse errors, or timeout, the matcher fails open—allowing the command but logging diagnostics. To maintain the zero-false-negative guarantee for destructive payloads, dcg implements fallback regex scanners targeting the most dangerous sinks (exec-style functions, filesystem-deleting calls). These backstops catch cases where the AST matcher cannot execute fast enough or the language lacks support.

Integration with the Heredoc Pipeline

The orchestration logic in src/evaluator.rs wires the tiers together:

1. Tier 1 (check_triggers) identifies potential heredocs or inline scripts.
2. Tier 2 (extract_content) produces ExtractedContent { language, content, … } structures.
3. Tier 3 (AstMatcher::find_matches) receives the extracted content, selects the appropriate language grammar, and executes the AST match.

If any returned PatternMatch carries Critical or High severity, dcg denies the original shell command and returns a structured JSON denial with the specific rule ID, line number, and remediation suggestion.

Why AST-Based Matching Matters

Precision: By parsing scripts into syntax trees, the matcher distinguishes executable code from comments, string literals, and documentation, eliminating the false positives inherent in pure regex approaches.

Speed: The parser executes only after Tier 2 extraction filters out non-script content, and the 20 ms worker thread timeout ensures typical scripts finish in under 5 ms.

Extensibility: New destructive patterns are declared as ast-grep rules in src/config.rs (e.g., detecting execSync calls or subprocess.run invocations) without modifying the surrounding Rust logic, allowing rapid updates to detection capabilities.

Practical Implementation Example

The following Rust example demonstrates the complete flow from trigger detection to AST matching:

use destructive_command_guard::heredoc::{check_triggers, extract_content, ExtractionLimits};
use destructive_command_guard::ast_matcher::AstMatcher;

// Tier 1: Fast trigger detection
let cmd = "python3 -c 'import os; os.system(\"rm -rf /\")'";
assert_eq!(check_triggers(cmd), TriggerResult::Triggered);

// Tier 2: Extract the inline script with resource limits
let limits = ExtractionLimits::default();
let extraction = extract_content(cmd, &limits);

if let ExtractionResult::Extracted(contents) = extraction {
    // Tier 3: Run the AST matcher on extracted Python code
    let matcher = AstMatcher::default();
    for ec in contents {
        if let Some(block) = matcher.has_blocking_match(&ec.content, ec.language) {
            println!("Blocked destructive command: {}", block.reason);
            println!("Suggestion: {}", block.suggestion);
        }
    }
}

Summary

• The AST matcher in src/ast_matcher.rs provides the third tier of detection after regex triggers and content extraction in src/heredoc.rs.
• AstMatcher::find_matches runs in a dedicated worker thread with a 20 ms timeout to prevent blocking interactive shells.
• Pattern matching uses actual syntax trees via ast-grep, distinguishing executable code from comments and string literals through refinement functions like refine_python_match.
• The system fails open on parse errors or timeouts, with fallback regex scanners ensuring dangerous commands still trigger blocks.
• New patterns are configured declaratively in src/config.rs and compiled at initialization without requiring changes to the core matching logic.

Frequently Asked Questions

What languages does the AST matcher support?

The matcher supports Python, JavaScript, TypeScript, Ruby, Bash, Go, and PHP through native ast-grep integrations. Perl requires special handling via precompile_perl_patterns() due to parsing complexity, while unsupported languages trigger the fallback regex scanners.

How does the matcher handle timeouts?

The implementation spawns a dedicated worker thread for each AST parsing operation and sets a shared AtomicBool flag. If parsing exceeds the 20 ms budget, the parent thread sets this flag, causing the worker to terminate immediately and return a timeout error, ensuring the shell remains responsive.

Can the AST matcher detect destructive commands inside string literals?

No, by design the refinement phase filters out matches occurring within string literals or comments. For example, rm -rf / inside a Python docstring is ignored, while os.system("rm -rf /") in executable code is flagged. This precision eliminates false positives from documentation or data strings.

Where are the destructive patterns defined?

Destructive patterns are declared in src/config.rs as declarative rules that map to ast-grep patterns. During AstMatcher initialization, these rules are pre-compiled into PrecompiledPattern structs and organized by language in a HashMap<ScriptLanguage, Vec<PrecompiledPattern>>, enabling O(1) lookup during matching operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}