How Wigolo Fetch Tier Escalation Works: From Plain HTTP to Headless Browser

Wigolo always attempts the cheap plain-HTTP tier first and only escalates to the expensive headless-browser tier when the page requires JavaScript, returns a shell capture, or produces a plain-text error body.

The fetch tool in KnockOutEZ/wigolo implements an intelligent wigolo fetch tier escalation strategy that balances cost efficiency with content completeness. By starting with lightweight HTTP requests and conditionally upgrading to Playwright-driven browser automation, the system minimizes resource consumption while ensuring dynamic web applications are fully rendered.

The Tier Escalation Pipeline

The escalation logic resides in the SmartRouter invoked by handleFetch in src/tools/fetch.ts. The following steps describe the exact decision flow implemented in the source code.

Step 1: Mode Resolution and SSRF Protection

Every fetch request begins with validation and mode resolution. The resolveMode function in src/util/mode.ts (lines 13-17) processes the requested mode—default, cache, stealth, or others—while validateFetchUrl in src/tools/fetch.ts (lines 31-68) and guardFetchUrl in src/watch/ssrf.ts (lines 85-100) enforce URL syntax and SSRF safety before any network tier is engaged.

Step 2: Cache Shortcut

When the mode is not stealth and force_refresh is false, Wigolo checks for cached content via getCachedContent → isCacheUsable in src/tools/fetch.ts (lines 7-12). A fresh, non-shell cache entry returns immediately, bypassing both the HTTP and browser tiers entirely.

Step 3: Plain-HTTP Tier Attempt

On cache miss, the router receives the fetch request with options including {render_js: "auto", …}. The first attempt uses the plain-HTTP tier, a fast TLS request that never spawns a browser. This tier returns a raw result containing method: "http" and notably no contentCompleteness field, as seen in the router.fetch call in src/tools/fetch.ts (lines 44-51).

Step 4: Escalation Triggers

The SmartRouter escalates to the Playwright tier when any of three conditions are met:

• JavaScript Required: The caller explicitly sets render_js: "always" or the router detects that JavaScript is necessary to generate meaningful markup.
• Shell Capture: The HTTP response contains only a minimal HTML skeleton, indicated by contentCompleteness.level === "shell".
• Plain-Text Error Bodies: The HTTP tier returns a 4xx/5xx status with a body whose Content-Type is plain-text, Markdown, JSON, or XML. The router treats this as a failure and retries with the browser tier to capture richer error pages.

This decision logic is handled within the router and observed in handleFetch in src/tools/fetch.ts (lines 66-68) through inspection of raw.contentCompleteness.

Step 5: Headless Browser Execution

Upon escalation, the system invokes fetchWithPlaywright in src/fetch/playwright-tier.ts (lines 66-71). This tier spawns a Playwright instance, loads the target URL, optionally executes user-provided actions (clicks, scrolls, etc.), and returns a result with method: "browser" and a contentCompleteness object describing the render quality (full, shell, etc.).

Step 6: Result Normalization and Auditability

After obtaining the raw tier result, handleFetch constructs the public FetchOutput. It copies the router-chosen method into fetch_method via out.fetch_method = raw.method in src/tools/fetch.ts (lines 63-66), allowing callers to audit which tier served the content. For browser-tier results, the content_completeness label is also propagated (lines 66-68).

Stealth Mode Bypass

When mode: "stealth" is requested, the logic in src/tools/fetch.ts (lines 4-7) forces a fresh fetch with no cache lookup and treats any browser-tier failure as a hard error. This prevents anti-bot blocks from being masked by stale cached shells.

Configuring Fetch Behavior

Developers control the wigolo fetch tier escalation behavior through specific parameters that influence the router's tier selection.

Automatic vs Forced JavaScript Rendering

Setting render_js: "auto" allows the SmartRouter to decide whether JavaScript is needed based on initial HTTP responses. Conversely, render_js: "always" bypasses the HTTP tier completely, forcing immediate Playwright execution for sites known to require heavy client-side rendering.

Stealth Mode for Anti-Bot Protection

Use mode: "stealth" when fetching from sites with aggressive bot detection. This guarantees a fresh browser context with no cache fallback, ensuring that blocks or CAPTCHAs are captured accurately rather than returning a cached shell or HTTP error.

Practical Code Examples

The following examples demonstrate how to interact with the tier escalation system:

// 1. Simple fetch - tries HTTP first, escalates only if needed
const result = await wigolo.fetch({
  url: 'https://example.com/docs',
  render_js: 'auto',          // "auto" = use browser only when required
});
console.log(result.fetch_method);   // "http" (or "browser" after escalation)

// 2. Force full JS rendering - bypasses the HTTP tier completely
const jsResult = await wigolo.fetch({
  url: 'https://react-app.example',
  render_js: 'always',        // forces Playwright tier
});
console.log(jsResult.fetch_method); // "browser"

// 3. Stealth mode - always fresh, never uses cached shells
const stealth = await wigolo.fetch({
  url: 'https://private-site.internal',
  mode: 'stealth',
});

Summary

• Fast-first strategy: The plain-HTTP tier in src/tools/fetch.ts is always attempted first unless force_refresh or stealth mode dictates otherwise.
• Conditional escalation: The SmartRouter upgrades to the Playwright tier in src/fetch/playwright-tier.ts only for JavaScript requirements, shell captures, or plain-text error responses.
• Full auditability: The fetch_method field in the output reveals which tier served the request ("http" or "browser"), while content_completeness indicates render quality for browser results.
• Cache awareness: In mode: "cache", the system never escalates to the browser tier, returning a cache_miss error instead of spawning Playwright.

Frequently Asked Questions

What triggers wigolo to switch from HTTP to headless browser?

Wigolo escalates from the HTTP tier to the headless-browser tier when the render_js parameter is set to "always", when the HTTP response indicates a shell capture (contentCompleteness.level === "shell"), or when the HTTP tier returns a plain-text error body (4xx/5xx). The SmartRouter evaluates these conditions after the initial router.fetch call in src/tools/fetch.ts.

How can I force the browser tier to always execute?

Pass render_js: "always" in your fetch options. This configuration bypasses the plain-HTTP tier entirely and immediately invokes fetchWithPlaywright from src/fetch/playwright-tier.ts, ensuring full JavaScript execution regardless of the initial HTTP response.

What is the difference between default mode and stealth mode?

Default mode allows cache hits and graceful degradation, potentially returning cached content or HTTP-tier results. Stealth mode (mode: "stealth"), implemented in src/tools/fetch.ts (lines 4-7), forces a fresh fetch with no cache lookup and treats browser-tier failures as hard errors, making it ideal for auditing anti-bot protections or ensuring completely fresh renders.

Does wigolo cache headless browser results?

Yes, but with safeguards. The isCacheUsable check in src/tools/fetch.ts (lines 7-12) considers cached browser results valid only if they are not shell captures and the mode is not stealth. This prevents incomplete renders from being served repeatedly while allowing fully rendered pages to benefit from caching.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}