How the Variable Extraction and Management System Works in Prompt-Optimizer

The variable extraction and management system in prompt-optimizer uses a two-layer architecture: a pure regex-driven extraction layer for parsing {{variable}} placeholders and a manager interface that handles CRUD operations, validation, and deterministic hashing for cacheability.

The linshenkx/prompt-optimizer repository implements a robust variable extraction and management system that treats template placeholders as first-class entities. This system ensures that every {{variable}} tag is detected, validated, and tracked throughout the prompt lifecycle, preventing runtime errors and enabling efficient caching mechanisms.

Core Architecture of the Variable Extraction System

The variable extraction and management system splits responsibilities across two distinct layers to maintain separation of concerns between parsing logic and state management.

The Extraction and Validation Layer

This layer handles the detection of {{variable}} syntax, filters out Mustache control tags like {{#if}} or {{>partial}}, and enforces strict naming conventions. The core implementation resides in packages/ui/src/utils/prompt-variables.ts and packages/ui/src/types/variable.ts, providing pure functions that never mutate the original template.

The Management API Layer

The IVariableManager interface, defined in packages/ui/src/types/variable.ts, abstracts storage mechanisms behind a consistent CRUD contract. Concrete implementations in the UI layer (typically Pinia stores) handle persistence to localStorage or IndexedDB, while the interface ensures that variable resolution, hashing, and missing-variable detection remain deterministic across the application.

How Variable Extraction Works in Practice

The extraction engine uses a carefully crafted regular expression to identify valid variable placeholders while rejecting illegal syntax.

The Regular Expression Engine

The system uses the following pattern defined in packages/ui/src/utils/prompt-variables.ts:

const VARIABLE_PATTERN = /\{\{\s*([^\d{}\s][^{}\s]*)\s*\}\}/gu;

This regex matches {{foo}} or {{ foo }} while explicitly rejecting names that start with digits or contain whitespace. The u flag ensures Unicode compatibility, and the g flag enables global matching across multi-line templates.

Validation Rules and Forbidden Syntax

The VARIABLE_VALIDATION constant in packages/ui/src/types/variable.ts enforces additional constraints:

export const VARIABLE_VALIDATION = {
  NAME_PATTERN: /^[^\s{}]+$/,                     // no spaces or braces
  NO_NUMBER_START_PATTERN: /^\d/u,                // cannot start with a digit
  FORBIDDEN_PREFIX_PATTERN: /^[#/^!>&]/u,         // blocks Mustache control prefixes
  RESERVED_NAMES: ['__proto__','prototype','constructor'] as const,
  MAX_NAME_LENGTH: 50,
  MAX_VALUE_LENGTH: 10000,
  VARIABLE_SCAN_PATTERN: /\{\{\s*([^\d{}\s][^{}\s]*)\s*\}\}/g
} as const;

The helper isValidVariableName(name) returns true only when a name passes all checks, including the forbidden prefix pattern that filters out Mustache control tags like {{#if}} or {{/if}}.

Core Utility Functions

The extraction layer exposes several pure utility functions in packages/ui/src/utils/prompt-variables.ts:

• scanVariableNames(content): Returns a deduplicated array of valid variable names found in a string.
• replaceVariablesInContent(content, variables): Substitutes placeholders with supplied values, leaving placeholders untouched when variables are missing.
• findMissingVariables(content, variables): Lists variables that appear in the template but have no value or an empty string.
• findForbiddenTemplateSyntax(content): Detects illegal constructs such as triple braces ({{{…}}}) or unescaped ampersands ({{&…}}).
• buildPromptExecutionContext(content, variables): Bundles hash, missing vars, rendered content, and forbidden syntax into a single object for downstream components.
• buildConversationExecutionContext(messages, variables): Performs the same bundling for arrays of messages in multi-turn conversations.

The Variable Management Interface

The IVariableManager interface provides a unified API for variable lifecycle management, abstracting storage details from the UI components.

CRUD Operations and Resolution

The interface supports standard CRUD operations:

export interface IVariableManager {
  setVariable(name: string, value: string): void;
  getVariable(name: string): string | undefined;
  deleteVariable(name: string): void;
  listVariables(): Record<string, string>;
  
  // Resolution merges predefined + custom variables
  resolveAllVariables(context?: Record<string, unknown>): Record<string, string>;
  
  // Validation and scanning
  validateVariableName(name: string): boolean;
  scanVariablesInContent(content: string): string[];
  
  // Source tracking
  getVariableSource(name: string): VariableSource;
  isPredefinedVariable(name: string): boolean;
  
  // Advanced mode and conversation helpers
  getAdvancedModeEnabled(): boolean;
  setAdvancedModeEnabled(enabled: boolean): void;
  getLastConversationMessages(): ConversationMessage[];
  setLastConversationMessages(messages: ConversationMessage[]): void;
  
  // Statistics and import/export
  getStatistics(): { customVariableCount: number; predefinedVariableCount: number; totalVariableCount: number; advancedModeEnabled: boolean; };
  replaceVariables(content: string, variables?: Record<string, string>): string;
  detectMissingVariables(content: string | ConversationMessage[], availableVariables?: Record<string, string>): string[];
  exportVariables(): string;
  importVariables(jsonData: string): void;
}

The resolveAllVariables method is particularly important: it merges predefined variables imported from @prompt-optimizer/core with user-defined values, ensuring that system-level placeholders like {{date}} are always available without manual configuration.

Hashing and Change Detection

The manager tracks a hash of the variable set (via hashVariables, implied by the cacheability requirements) to enable cheap change detection. This allows the optimizer to determine whether a prompt needs to be re-run based on whether variable values have changed since the last execution, preventing unnecessary API calls and improving performance.

Runtime Flow: From Template to Rendered Prompt

The variable extraction and management system follows a deterministic pipeline that guarantees validation before any LLM API calls occur:

1. Template editing: When a user edits a prompt, the UI component calls scanVariableNames to highlight variable placeholders in real-time.
2. Variable inspection: Opening the Variable Manager triggers listVariables() to fetch current values and scanVariablesInContent to identify required placeholders.
3. Value updates: Calling setVariable(name, value) updates the Pinia store, persists to localStorage or IndexedDB, and recomputes the variablesHash for change detection.
4. Pre-flight validation: Before sending to the backend, the UI builds a PromptExecutionContext via buildPromptExecutionContext(content, variables), which bundles the rendered content, missing variables, and forbidden syntax detection.
5. Backend processing: The backend receives renderedContent with placeholders already replaced, along with metadata about missingVariables and forbiddenTemplateSyntax for final validation.

This pipeline ensures that all placeholders are validated early, missing variables surface to users before API calls, and illegal Mustache syntax is caught to prevent runtime template errors.

Code Examples

Extracting Variables from Raw Prompts

To extract all valid variable names from a template string:

import { scanVariableNames } from '@/utils/prompt-variables';

const raw = `请根据 {{任务描述}} 为 {{目标用户}} 编写 {{文档类型}}，要求 {{质量要求}}。`;
const vars = scanVariableNames(raw);
// vars => ['任务描述', '目标用户', '文档类型', '质量要求']

See the implementation in scanVariableNames.

Replacing Values in Templates

To substitute placeholders with actual values while preserving unmatched placeholders:

import { replaceVariablesInContent } from '@/utils/prompt-variables';

const template = `Hello {{ name }}, today is {{ date }}.`;
const values = { name: 'Alice', date: '2024‑10‑01' };
const rendered = replaceVariablesInContent(template, values);
// rendered => "Hello Alice, today is 2024‑10‑01."

See the implementation in replaceVariablesInContent.

Building Execution Contexts

To create a complete execution context with validation metadata:

import { buildPromptExecutionContext } from '@/utils/prompt-variables';

const ctx = buildPromptExecutionContext(template, values);
console.log(ctx.missingVariables);      // [] – all supplied
console.log(ctx.forbiddenTemplateSyntax); // [] – no illegal syntax
console.log(ctx.renderedContent);       // rendered string ready for the LLM

See the implementation in buildPromptExecutionContext.

Using the Variable Manager

To interact with the variable management API in UI components:

import { useVariableManager } from '@/stores/variable';

const mgr = useVariableManager();

// Add a variable
mgr.setVariable('projectName', 'Prompt‑Optimizer');

// List all
console.log(mgr.listVariables());
// => { projectName: 'Prompt‑Optimizer', … }

// Resolve for a prompt
const resolved = mgr.resolveAllVariables();

The store implements IVariableManager; see the interface definition in IVariableManager.

Key Source Files

File	Role
packages/ui/src/utils/prompt-variables.ts	Core regex, extraction, replacement, and context-building utilities.
packages/ui/src/types/variable.ts	Type definitions, validation constants, and the IVariableManager contract.
packages/ui/src/stores/variable/... (Pinia store)	Concrete implementation of the manager, persistence, and UI bindings.
packages/ui/src/components/VariableExtraction/*	UI hooks that call scanVariableNames and display highlights in the editor.
@prompt-optimizer/core	Supplies predefined system-level variables used by resolveAllVariables.

Summary

• Two-layer architecture: The variable extraction and management system separates pure extraction logic (prompt-variables.ts) from stateful management (IVariableManager interface).
• Strict validation: Variables must match /\{\{\s*([^\d{}\s][^{}\s]*)\s*\}\}/gu and pass naming rules that block Mustache control tags and reserved JavaScript property names.
• Deterministic resolution: The resolveAllVariables method merges user-defined variables with predefined system variables from @prompt-optimizer/core, ensuring consistent availability of placeholders like {{date}}.
• Pre-flight safety: buildPromptExecutionContext bundles rendered content, missing variable detection, and forbidden syntax checking before any LLM API calls occur.
• Change detection: The manager computes hashes of variable sets to enable cacheability and prevent unnecessary re-execution of optimized prompts.

Frequently Asked Questions

How does prompt-optimizer prevent Mustache control tags from being treated as variables?

The validation layer in packages/ui/src/types/variable.ts defines a FORBIDDEN_PREFIX_PATTERN of /^[#/^!>&]/u that detects Mustache control prefixes like #, /, ^, !, >, and &. When scanVariableNames processes content, it filters out any tags matching these prefixes, ensuring only pure variable placeholders are extracted while control flow syntax is ignored.

What happens if a template references a variable that hasn't been defined?

The findMissingVariables utility in packages/ui/src/utils/prompt-variables.ts compares variables detected in the template against the supplied values map. When buildPromptExecutionContext is called, it returns a missingVariables array containing the names of any placeholders lacking values. The UI layer surfaces these to the user before sending requests to the LLM, preventing incomplete prompts from executing.

Can the variable management system handle multi-turn conversations?

Yes, the system provides buildConversationExecutionContext in packages/ui/src/utils/prompt-variables.ts, which processes arrays of ConversationMessage objects. This function applies the same extraction, validation, and replacement logic across all messages in a conversation thread, ensuring consistent variable resolution and missing-variable detection throughout multi-turn interactions managed by the IVariableManager interface.

How does the system optimize performance when variables haven't changed?

The IVariableManager interface includes hashing capabilities (referenced in the resolveAllVariables context and statistics tracking) that compute a deterministic hash of the current variable set. This hash enables the optimizer to compare the current state against cached executions. When variables remain unchanged, the system can retrieve previous optimization results without re-running the LLM, significantly reducing API costs and latency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}