# How the Variable Extraction and Management System Works in Prompt-Optimizer > Discover how prompt-optimizer's variable extraction and management system leverages a two-layer architecture for efficient CRUD operations, validation, and cacheable hashing. - Repository: [且炼时光/prompt-optimizer](https://github.com/linshenkx/prompt-optimizer) - Tags: internals - Published: 2026-02-23 --- **The variable extraction and management system in prompt-optimizer uses a two-layer architecture: a pure regex-driven extraction layer for parsing `{{variable}}` placeholders and a manager interface that handles CRUD operations, validation, and deterministic hashing for cacheability.** The `linshenkx/prompt-optimizer` repository implements a robust variable extraction and management system that treats template placeholders as first-class entities. This system ensures that every `{{variable}}` tag is detected, validated, and tracked throughout the prompt lifecycle, preventing runtime errors and enabling efficient caching mechanisms. ## Core Architecture of the Variable Extraction System The variable extraction and management system splits responsibilities across two distinct layers to maintain separation of concerns between parsing logic and state management. ### The Extraction and Validation Layer This layer handles the detection of `{{variable}}` syntax, filters out Mustache control tags like `{{#if}}` or `{{>partial}}`, and enforces strict naming conventions. The core implementation resides in [`packages/ui/src/utils/prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/utils/prompt-variables.ts) and [`packages/ui/src/types/variable.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/types/variable.ts), providing pure functions that never mutate the original template. ### The Management API Layer The `IVariableManager` interface, defined in [`packages/ui/src/types/variable.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/types/variable.ts), abstracts storage mechanisms behind a consistent CRUD contract. Concrete implementations in the UI layer (typically Pinia stores) handle persistence to `localStorage` or IndexedDB, while the interface ensures that variable resolution, hashing, and missing-variable detection remain deterministic across the application. ## How Variable Extraction Works in Practice The extraction engine uses a carefully crafted regular expression to identify valid variable placeholders while rejecting illegal syntax. ### The Regular Expression Engine The system uses the following pattern defined in [`packages/ui/src/utils/prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/utils/prompt-variables.ts): ```typescript const VARIABLE_PATTERN = /\{\{\s*([^\d{}\s][^{}\s]*)\s*\}\}/gu; ``` This regex matches `{{foo}}` or `{{ foo }}` while explicitly rejecting names that start with digits or contain whitespace. The `u` flag ensures Unicode compatibility, and the `g` flag enables global matching across multi-line templates. ### Validation Rules and Forbidden Syntax The `VARIABLE_VALIDATION` constant in [`packages/ui/src/types/variable.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/types/variable.ts) enforces additional constraints: ```typescript export const VARIABLE_VALIDATION = { NAME_PATTERN: /^[^\s{}]+$/, // no spaces or braces NO_NUMBER_START_PATTERN: /^\d/u, // cannot start with a digit FORBIDDEN_PREFIX_PATTERN: /^[#/^!>&]/u, // blocks Mustache control prefixes RESERVED_NAMES: ['__proto__','prototype','constructor'] as const, MAX_NAME_LENGTH: 50, MAX_VALUE_LENGTH: 10000, VARIABLE_SCAN_PATTERN: /\{\{\s*([^\d{}\s][^{}\s]*)\s*\}\}/g } as const; ``` The helper `isValidVariableName(name)` returns `true` only when a name passes all checks, including the forbidden prefix pattern that filters out Mustache control tags like `{{#if}}` or `{{/if}}`. ### Core Utility Functions The extraction layer exposes several pure utility functions in [`packages/ui/src/utils/prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/utils/prompt-variables.ts): - **`scanVariableNames(content)`**: Returns a deduplicated array of valid variable names found in a string. - **`replaceVariablesInContent(content, variables)`**: Substitutes placeholders with supplied values, leaving placeholders untouched when variables are missing. - **`findMissingVariables(content, variables)`**: Lists variables that appear in the template but have no value or an empty string. - **`findForbiddenTemplateSyntax(content)`**: Detects illegal constructs such as triple braces (`{{{…}}}`) or unescaped ampersands (`{{&…}}`). - **`buildPromptExecutionContext(content, variables)`**: Bundles hash, missing vars, rendered content, and forbidden syntax into a single object for downstream components. - **`buildConversationExecutionContext(messages, variables)`**: Performs the same bundling for arrays of messages in multi-turn conversations. ## The Variable Management Interface The `IVariableManager` interface provides a unified API for variable lifecycle management, abstracting storage details from the UI components. ### CRUD Operations and Resolution The interface supports standard CRUD operations: ```typescript export interface IVariableManager { setVariable(name: string, value: string): void; getVariable(name: string): string | undefined; deleteVariable(name: string): void; listVariables(): Record; // Resolution merges predefined + custom variables resolveAllVariables(context?: Record): Record; // Validation and scanning validateVariableName(name: string): boolean; scanVariablesInContent(content: string): string[]; // Source tracking getVariableSource(name: string): VariableSource; isPredefinedVariable(name: string): boolean; // Advanced mode and conversation helpers getAdvancedModeEnabled(): boolean; setAdvancedModeEnabled(enabled: boolean): void; getLastConversationMessages(): ConversationMessage[]; setLastConversationMessages(messages: ConversationMessage[]): void; // Statistics and import/export getStatistics(): { customVariableCount: number; predefinedVariableCount: number; totalVariableCount: number; advancedModeEnabled: boolean; }; replaceVariables(content: string, variables?: Record): string; detectMissingVariables(content: string | ConversationMessage[], availableVariables?: Record): string[]; exportVariables(): string; importVariables(jsonData: string): void; } ``` The `resolveAllVariables` method is particularly important: it merges predefined variables imported from `@prompt-optimizer/core` with user-defined values, ensuring that system-level placeholders like `{{date}}` are always available without manual configuration. ### Hashing and Change Detection The manager tracks a hash of the variable set (via `hashVariables`, implied by the cacheability requirements) to enable cheap change detection. This allows the optimizer to determine whether a prompt needs to be re-run based on whether variable values have changed since the last execution, preventing unnecessary API calls and improving performance. ## Runtime Flow: From Template to Rendered Prompt The variable extraction and management system follows a deterministic pipeline that guarantees validation before any LLM API calls occur: 1. **Template editing**: When a user edits a prompt, the UI component calls `scanVariableNames` to highlight variable placeholders in real-time. 2. **Variable inspection**: Opening the Variable Manager triggers `listVariables()` to fetch current values and `scanVariablesInContent` to identify required placeholders. 3. **Value updates**: Calling `setVariable(name, value)` updates the Pinia store, persists to `localStorage` or IndexedDB, and recomputes the `variablesHash` for change detection. 4. **Pre-flight validation**: Before sending to the backend, the UI builds a `PromptExecutionContext` via `buildPromptExecutionContext(content, variables)`, which bundles the rendered content, missing variables, and forbidden syntax detection. 5. **Backend processing**: The backend receives `renderedContent` with placeholders already replaced, along with metadata about `missingVariables` and `forbiddenTemplateSyntax` for final validation. This pipeline ensures that all placeholders are validated early, missing variables surface to users before API calls, and illegal Mustache syntax is caught to prevent runtime template errors. ## Code Examples ### Extracting Variables from Raw Prompts To extract all valid variable names from a template string: ```typescript import { scanVariableNames } from '@/utils/prompt-variables'; const raw = `请根据 {{任务描述}} 为 {{目标用户}} 编写 {{文档类型}}，要求 {{质量要求}}。`; const vars = scanVariableNames(raw); // vars => ['任务描述', '目标用户', '文档类型', '质量要求'] ``` See the implementation in [`scanVariableNames`](https://github.com/linshenkx/prompt-optimizer/blob/develop/packages/ui/src/utils/prompt-variables.ts#L19-L33). ### Replacing Values in Templates To substitute placeholders with actual values while preserving unmatched placeholders: ```typescript import { replaceVariablesInContent } from '@/utils/prompt-variables'; const template = `Hello {{ name }}, today is {{ date }}.`; const values = { name: 'Alice', date: '2024‑10‑01' }; const rendered = replaceVariablesInContent(template, values); // rendered => "Hello Alice, today is 2024‑10‑01." ``` See the implementation in [`replaceVariablesInContent`](https://github.com/linshenkx/prompt-optimizer/blob/develop/packages/ui/src/utils/prompt-variables.ts#L60-L75). ### Building Execution Contexts To create a complete execution context with validation metadata: ```typescript import { buildPromptExecutionContext } from '@/utils/prompt-variables'; const ctx = buildPromptExecutionContext(template, values); console.log(ctx.missingVariables); // [] – all supplied console.log(ctx.forbiddenTemplateSyntax); // [] – no illegal syntax console.log(ctx.renderedContent); // rendered string ready for the LLM ``` See the implementation in [`buildPromptExecutionContext`](https://github.com/linshenkx/prompt-optimizer/blob/develop/packages/ui/src/utils/prompt-variables.ts#L107-L117). ### Using the Variable Manager To interact with the variable management API in UI components: ```typescript import { useVariableManager } from '@/stores/variable'; const mgr = useVariableManager(); // Add a variable mgr.setVariable('projectName', 'Prompt‑Optimizer'); // List all console.log(mgr.listVariables()); // => { projectName: 'Prompt‑Optimizer', … } // Resolve for a prompt const resolved = mgr.resolveAllVariables(); ``` The store implements `IVariableManager`; see the interface definition in [`IVariableManager`](https://github.com/linshenkx/prompt-optimizer/blob/develop/packages/ui/src/types/variable.ts#L54-L81). ## Key Source Files | File | Role | |------|------| | **[`packages/ui/src/utils/prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/utils/prompt-variables.ts)** | Core regex, extraction, replacement, and context-building utilities. | | **[`packages/ui/src/types/variable.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/types/variable.ts)** | Type definitions, validation constants, and the `IVariableManager` contract. | | **`packages/ui/src/stores/variable/...`** (Pinia store) | Concrete implementation of the manager, persistence, and UI bindings. | | **`packages/ui/src/components/VariableExtraction/*`** | UI hooks that call `scanVariableNames` and display highlights in the editor. | | **`@prompt-optimizer/core`** | Supplies predefined system-level variables used by `resolveAllVariables`. | ## Summary - **Two-layer architecture**: The variable extraction and management system separates pure extraction logic ([`prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/prompt-variables.ts)) from stateful management (`IVariableManager` interface). - **Strict validation**: Variables must match `/\{\{\s*([^\d{}\s][^{}\s]*)\s*\}\}/gu` and pass naming rules that block Mustache control tags and reserved JavaScript property names. - **Deterministic resolution**: The `resolveAllVariables` method merges user-defined variables with predefined system variables from `@prompt-optimizer/core`, ensuring consistent availability of placeholders like `{{date}}`. - **Pre-flight safety**: `buildPromptExecutionContext` bundles rendered content, missing variable detection, and forbidden syntax checking before any LLM API calls occur. - **Change detection**: The manager computes hashes of variable sets to enable cacheability and prevent unnecessary re-execution of optimized prompts. ## Frequently Asked Questions ### How does prompt-optimizer prevent Mustache control tags from being treated as variables? The validation layer in [`packages/ui/src/types/variable.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/types/variable.ts) defines a `FORBIDDEN_PREFIX_PATTERN` of `/^[#/^!>&]/u` that detects Mustache control prefixes like `#`, `/`, `^`, `!`, `>`, and `&`. When `scanVariableNames` processes content, it filters out any tags matching these prefixes, ensuring only pure variable placeholders are extracted while control flow syntax is ignored. ### What happens if a template references a variable that hasn't been defined? The `findMissingVariables` utility in [`packages/ui/src/utils/prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/utils/prompt-variables.ts) compares variables detected in the template against the supplied values map. When `buildPromptExecutionContext` is called, it returns a `missingVariables` array containing the names of any placeholders lacking values. The UI layer surfaces these to the user before sending requests to the LLM, preventing incomplete prompts from executing. ### Can the variable management system handle multi-turn conversations? Yes, the system provides `buildConversationExecutionContext` in [`packages/ui/src/utils/prompt-variables.ts`](https://github.com/linshenkx/prompt-optimizer/blob/main/packages/ui/src/utils/prompt-variables.ts), which processes arrays of `ConversationMessage` objects. This function applies the same extraction, validation, and replacement logic across all messages in a conversation thread, ensuring consistent variable resolution and missing-variable detection throughout multi-turn interactions managed by the `IVariableManager` interface. ### How does the system optimize performance when variables haven't changed? The `IVariableManager` interface includes hashing capabilities (referenced in the `resolveAllVariables` context and statistics tracking) that compute a deterministic hash of the current variable set. This hash enables the optimizer to compare the current state against cached executions. When variables remain unchanged, the system can retrieve previous optimization results without re-running the LLM, significantly reducing API costs and latency.