How to Perform DSL VM / Custom Opcode VM Reverse Engineering

DSL VM reverse engineering involves identifying characteristic JavaScript patterns such as IIFE entry points with single-letter variables, extracting opcode mappings from the central interpreter function, and applying a six-phase static analysis workflow to reconstruct the original semantics.

The reverse-skill repository provides a comprehensive methodology for analyzing JavaScript-based domain-specific language (DSL) virtual machines commonly found in web risk-control and captcha systems. This guide walks through the complete process of DSL VM / custom opcode VM reverse engineering, from initial detection to runtime capture, using the techniques documented in skills/reverse-engineering/dsl-vm-reverse/SKILL.md.

Identifying DSL VM Patterns in JavaScript

Before analysis begins, you must confirm the target script implements a custom opcode VM rather than standard JavaScript or WebAssembly.

Characteristic IIFE Structures

DSL VMs typically expose an immediately-invoked function expression (IIFE) entry point containing dense variable declarations using single-letter identifiers. According to the source analysis in SKILL.md#L30-L34, look for initialization patterns such as var U=void 0, y=parseInt, … where short variable names map to built-in functions or numeric constants. This obfuscation pattern indicates the presence of a bytecode interpreter rather than standard source code.

The Central Interpreter Loop

The VM core resides in a central dispatcher function—commonly named DG in the analyzed samples—that implements a switch-case loop dispatching opcodes. As documented in SKILL.md#L60-L70, this function reads encoded instructions from a bytecode array and routes execution to handler cases based on numeric opcode values. Locating this dispatcher confirms you are dealing with a register-based or stack-based virtual machine rather than native JavaScript.

Constant Table Conventions

These systems rely on a centralized constant pool, frequently referenced as C[9], which stores functions, strings, and literals accessed via numeric indexes. The SKILL.md#L74-L78 analysis notes that tracking references to C[9][index] reveals the VM's imported external interfaces and string table layout.

Six-Phase Static Analysis Workflow

Once identification confirms a DSL VM, apply the repeatable workflow defined in the reverse-skill documentation.

Phase 1: File Classification

Perform quick sanity checks to filter out standard WASM binaries (checking for magic bytes \0asm) and verify zero-byte ratios. Confirm the IIFE pattern and single-letter variable density in the first 2 KB of the script.

Phase 2: Variable-Mapping Extraction

Parse the initialization section to build a map of obfuscated variable names to their numeric constant assignments. This mapping decodes the symbolic references used throughout the bytecode.

Phase 3: Opcode Extraction and Classification

Use regular expressions to collect all case <num>: branches from the dispatcher function. Heuristically label each opcode based on surrounding code patterns—identifying BRANCH, CALL, ARITH, ALLOC, and other operation types.

Phase 4: Constant-Table (C[9]) Analysis

Enumerate all indices accessed via C[9][...] and inspect the surrounding context to understand data layout. This table typically contains the VM's external API hooks and cryptographic constants.

Phase 5: Export-Function Tracing

Follow registration calls such as AWSCInner.register() to locate public API entry points like getToken that ultimately invoke the VM. This bridges the gap between the obfuscated internals and the exposed functionality.

Phase 6: Runtime Capture

When static analysis proves insufficient, instantiate the VM in a controlled environment. Use Playwright, Selenium CDP, or pure Node.js to execute the script and capture runtime state, including decrypted bytecode or computed results.

Decoding Opcode Semantics and Bit Layout

Understanding the instruction encoding is critical for reconstructing high-level logic. According to SKILL.md#L37-L65, the VM encodes each instruction as a 32-bit integer with the following layout:

• Bits 0-4: Primary opcode (0-31)
• Bits 5-9: Sub-operation or register index (0-31)
• Bits 10-31: Immediate operand or offset value

Decoding uses simple bitwise masks as implemented in the reference:

aE = d[7] & 31;        // Extract opcode from lower 5 bits
O = d[7] >> 5 & 31;    // Extract sub-operation from next 5 bits

A complete reference table mapping numeric case values (0-25) to high-level operations—such as conditional branches, arithmetic, and memory allocation—is provided in the skill documentation. By correlating these numeric values with the decoding logic, analysts can reconstruct the original DSL semantics.

Practical Implementation Examples

The reverse-skill repository includes automation scripts for each analysis phase.

Detecting DSL VM Patterns

This Python script identifies the characteristic IIFE and variable initialization patterns:

import re, sys, pathlib

script = pathlib.Path('target.js').read_bytes()
head = script[:100]

if head.startswith(b'!f'):                     # IIFE start

    if b'var U=void 0' in head or b'U=void 0,y=parseInt' in head:
        print('→ DSL VM detected')
        sys.exit(0)

print('Not a DSL VM')

Extracting Opcode Lists

Use regex to enumerate all dispatcher cases:

import re, pathlib
s = pathlib.Path('target.js').read_text(errors='replace')

cases = sorted({int(m) for m in re.findall(r'case\s+(\d+):', s)})
print(f'Found {len(cases)} opcodes: {cases}')

Mapping Obfuscated Variables

Extract the initial variable mappings from the script header:

import re, pathlib
head = pathlib.Path('target.js').read_text(errors='replace')[:2000]
mappings = re.findall(r'var\s+(\w+)\s*=\s*(\d+)', head)
for name, val in mappings:
    print(f'{name:5} = {int(val):3d} (0x{int(val):02x})')

Runtime Capture with Playwright

For dynamic analysis, inject the VM into a browser context to capture execution results:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({headless: false});
  const page = await browser.newPage();

  await page.goto('https://example.com/vm.html');

  await page.waitForFunction(() =>
    window.AWSCInner && window.AWSCInner._modules && window.AWSCInner._modules['fy']
  );

  const token = await page.evaluate(() =>
    window.AWSCInner._modules['fy'].getToken({})
  );
  console.log('Captured token:', token);
  await browser.close();
})();

Summary

• DSL VMs in JavaScript are identified by IIFE entry points with single-letter variables, central dispatcher functions (commonly DG), and constant tables like C[9].
• The six-phase workflow progresses from file classification through variable mapping, opcode extraction, constant analysis, export tracing, and optional runtime capture.
• Opcodes follow a 32-bit encoding scheme where bits 0-4 define the primary operation, bits 5-9 define sub-operations, and the remaining bits carry immediate values.
• The reverse-skill methodology, documented in skills/reverse-engineering/dsl-vm-reverse/SKILL.md and demonstrated in the fireyejs.js case study (skills/field-journal/2026-07-05_dsl-vm-captcha-reverse.md), provides a repeatable framework for analyzing these systems.

Frequently Asked Questions

What is a DSL VM in the context of web security?

A domain-specific language virtual machine (DSL VM) is a bytecode interpreter embedded in JavaScript that executes custom instruction sets for risk-control, captcha, or data protection systems. These VMs obscure logic by compiling high-level operations into numeric opcodes processed by a central dispatcher loop.

How can I quickly identify a custom opcode VM in JavaScript?

Look for three specific patterns: an IIFE header with single-letter variable declarations (e.g., var U=void 0), a central switch-case dispatcher function (often named DG), and array-based constant lookups (typically C[9][index]). Automated detectors can scan for these signatures in the first 2 KB of a script.

What tools are required for DSL VM reverse engineering?

The methodology requires static analysis tools (Python with regex for parsing, text editors for code review) and dynamic analysis environments (Playwright, Selenium CDP, or Node.js) for runtime capture. No specialized decompilers are needed since the analysis relies on pattern matching and behavioral inspection.

How are opcodes typically encoded in these systems?

According to the reverse-skill source analysis, opcodes are encoded as 32-bit integers where the lowest 5 bits represent the primary operation, the next 5 bits represent sub-operations or registers, and the upper 22 bits contain immediate values or offsets. Decoding uses bitwise AND (& 31) and right-shift operations (>> 5 & 31) to extract these fields.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}