How Mako Ensures Security During Evaluation Runs with Its Defense‑in‑Depth Model

Mako protects evaluation runs through layered security controls including process isolation, capability‑based permissions, sandboxed tokens, and immutable audit logging.

Mako is an Apache‑hosted open‑source framework for safe, reproducible execution of AI‑driven evaluations. To prevent malicious code from escaping or corrupting results, Mako implements a defense‑in‑depth security model that stacks multiple independent safeguards. Each layer addresses a distinct threat vector, ensuring that a breach of any single control does not compromise the entire system.

Process Isolation: The Runtime Host Layer

The foundation of Mako's security architecture is strict process separation between the user interface and evaluation execution.

The Runtime Host (packages/runtime-host/src/runtime-host.ts) spawns a dedicated, sandboxed process for every evaluation run. This host process operates with its own memory space and does not share resources with the UI or other concurrent evaluations. If an evaluator contains malicious code, it remains confined to this isolated process.

Key implementation details from the source:

• Each evaluation receives a fresh process spawned by RuntimeHost
• The host enforces termination policies if security checks fail
• Process boundaries prevent memory corruption or credential leakage from the evaluator back to the user interface

Windows Sandbox: Low‑Integrity Tokens and Job Objects

On Windows platforms, Mako applies additional OS‑level restrictions through a restricted token sandbox documented in docs/architecture/windows-sandbox-rfc-v1.md.

The Windows implementation creates three protective mechanisms:

1. Restricted tokens — strip unnecessary privileges from the evaluation process
2. Low‑integrity labels — mark the process with a lower integrity level than standard user processes, preventing write access to protected system locations
3. Job objects — enforce resource limits (CPU, memory) and enable collective termination of all process trees associated with an evaluation

These controls limit what a compromised evaluator can do even if it escapes application‑level sandboxing.

Capability‑Based Permission System

Mako uses capability‑based access control to restrict what system resources evaluators can touch. The permission model is defined in packages/core/src/permissions.ts.

Every tool exposed to evaluators—Grep, Shell, ReadFile, WriteFile—performs explicit permission checks before executing any operation. The evaluator cannot directly access the filesystem, network, or operating system APIs. Instead, it must request specific capabilities through the PermissionContext:

// packages/core/src/permissions.ts pattern
export async function readFile(path: string, ctx: PermissionContext) {
  // Explicit permission check before any filesystem access
  await ctx.requirePermission({ type: 'fileRead', path });
  
  // Access performed only through sandboxed filesystem interface
  return await ctx.sandboxedFs.readFile(path, 'utf-8');
}

This design ensures that failures default to denial: if permission state is ambiguous, the system refuses the operation.

Immutable Audit Logging with Provenance Guarantees

Mako maintains an append‑only, tamper‑evident log of all evaluation activity to guarantee auditability and detect manipulation attempts.

The runtime records every model message, tool invocation, and permission decision as a RuntimeEvent. These events are written to runtime.sqlite using a write‑ahead log (WAL) with verification on read. A compromised evaluator cannot rewrite history because:

• The log is append‑only from the evaluator's perspective
• Cryptographic verification detects tampering
• The logging mechanism operates outside the evaluator's trust boundary

This provenance lock ensures that evaluation results remain forensically valid for compliance and debugging.

Secure Credential Storage Outside the Evaluator

Secrets such as API keys and access tokens are never exposed to the evaluator process.

Mako stores credentials in credential-vault.json, readable only by the OS account under which the Runtime Host operates. The UI receives only opaque references; actual secrets are injected at the moment the Runtime Host initiates a model API call. This out‑of‑band secret injection prevents credential theft even if an evaluator achieves arbitrary code execution.

Fail‑Closed Defaults and Automatic Termination

Mako's security posture is fail‑closed: any security check failure immediately halts evaluation rather than proceeding with degraded protection.

The SessionManager and Runtime Host enforce this policy:

// Conceptual pattern from runtime-host implementation
if (!securityToken.verify() || !sandboxBoundary.intact()) {
  // Immediate termination, no recovery attempt
  this.terminateEvaluation(TerminationReason.SECURITY_VIOLATION);
  throw new SecurityBoundaryError('Evaluation terminated: security check failed');
}

This prevents "soft failure" scenarios where a partially compromised system might continue operating.

Practical Implementation: Running a Sandboxed Evaluation

To execute an evaluation with full security controls enabled:

import { RuntimeHost } from '@maka/runtime-host';
import { EvalSpec } from '@maka/eval';

const host = new RuntimeHost({
  sandbox: true,               // Enable process isolation
  tokenPolicy: 'lowIntegrity', // Windows token restriction
  auditLog: 'runtime.sqlite',  // Immutable event logging
});

const spec: EvalSpec = {
  prompt: 'Analyze this codebase for security vulnerabilities',
  model: 'gpt-4o-mini',
  tools: ['grep', 'readFile'],  // Explicit capability grant
  permissions: {
    fileRead: ['/repo/**'],      // Scoped filesystem access
    network: false,              // No network access granted
  },
};

host.runEval(spec)
  .then(result => console.log('Completed:', result))
  .catch(err => {
    // Security violations surface here with detailed context
    console.error('Security boundary triggered:', err.message);
  });

The RuntimeHost configuration explicitly enables each security layer, making protection levels auditable in code review.

Summary

• Process isolation via RuntimeHost keeps evaluators in separate, disposable processes
• Windows sandboxing applies low‑integrity tokens and Job objects for OS‑level containment
• Capability‑based permissions in packages/core enforce least‑privilege access to all tools
• Immutable audit logging in runtime.sqlite provides tamper‑evident provenance
• Out‑of‑band credential storage prevents secret exposure to evaluator code
• Fail‑closed defaults terminate evaluations immediately when security checks fail

Frequently Asked Questions

What happens if an evaluator tries to escape its sandbox?

The Runtime Host terminates the process immediately. Windows evaluations with tokenPolicy: 'lowIntegrity' operate with severely restricted privileges even if code execution occurs, and Job objects limit resource consumption. The audit log records the termination event for forensic analysis.

How does Mako prevent credential theft by malicious evaluators?

Credentials reside in credential-vault.json outside the evaluator's filesystem view. The UI never receives raw secrets. Only the Runtime Host injects them during model API calls, and evaluators cannot access the vault location due to permission boundaries enforced before any tool operation.

Can evaluators modify their own audit logs to hide malicious behavior?

No. The runtime.sqlite log uses a write‑ahead log with verification on read. The logging mechanism operates in the Runtime Host process, separate from the evaluator. Even with arbitrary code execution, an evaluator cannot rewrite events it previously generated because the host maintains exclusive write access with integrity checking.

Is Mako's security model portable beyond Windows?

Yes. While the low‑integrity token and Job object mechanisms are Windows‑specific, the Runtime Host abstraction in packages/runtime-host supports pluggable sandbox backends. The capability‑based permission system and immutable logging operate identically across platforms, with platform‑specific sandbox implementations providing equivalent isolation guarantees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}