# How Wigolo Per-Domain Fetch Learning Works: Inspection and Reset Guide > Learn how Wigolo per-domain fetch learning optimizes strategies using TLS, cookies, and timers. Inspect and reset these settings with the wigolo tune CLI tool. - Repository: [Towhid Khan/wigolo](https://github.com/KnockOutEZ/wigolo) - Tags: how-to-guide - Published: 2026-07-29 --- **Wigolo automatically learns optimal fetch strategies for each domain by persisting TLS success counters, anti-bot clearance cookies, and backoff timers in a local SQLite database, which you can inspect or clear using the `wigolo tune` CLI command.** Wigolo's fetch router adapts its behavior per domain based on historical success and failure patterns. This learning system stores routing preferences, clearance tokens, and failure statistics in a SQLite table named `domain_routing`, enabling the tool to optimize between lightweight TLS impersonation and full browser automation. Understanding how this mechanism works—and how to reset it when troubleshooting—is essential for maintaining reliable data extraction pipelines. ## The SQLite Storage Architecture All per-domain learning data lives in a SQLite database located at `/wigolo.db` (default: `~/.wigolo/wigolo.db`). The schema is defined in migration [`008-antibot-clearance.sql`](https://github.com/KnockOutEZ/wigolo/blob/main/008-antibot-clearance.sql) and maintained through [`src/cache/store.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cache/store.ts). ```sql -- src/cache/migrations/008-antibot-clearance.sql CREATE TABLE domain_routing ( domain TEXT PRIMARY KEY, prefer_playwright INTEGER, http_failures INTEGER, prefer_tls_impersonation INTEGER, tls_success_count INTEGER, backoff_until TEXT, last_403_at TEXT, cf_clearance TEXT, clearance_ua TEXT, clearance_tier TEXT, clearance_expires_at TEXT, last_updated TEXT ); ``` This table tracks five key aspects of **wigolo's per-domain fetch learning**: - **TLS preference**: Whether to prefer TLS impersonation over browser automation - **Browser preference**: Whether Playwright should be the default tier - **Anti-bot clearance**: Cloudflare (or similar) clearance cookies with expiry and user agent - **Backoff timing**: Cool-down periods after repeated 403 responses - **Statistics**: Counters for TLS successes and HTTP failures ## How Learning Mechanisms Record Domain Behavior ### Tracking TLS Impersonation Success When the TLS-impersonation tier successfully fetches a URL, [`src/fetch/router.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/fetch/router.ts) calls `recordTlsImpersonationSuccess()` to increment a success counter. Once the counter reaches the configured threshold (defined by `tlsSuccessThreshold`), the system atomically sets `prefer_tls_impersonation` to `true` for that domain. ```ts // src/cache/store.ts export function recordTlsImpersonationSuccess(domain: string, threshold: number): DomainRoutingRow | null { const db = getDatabase(); const preferOnInsert = threshold <= 1 ? 1 : 0; db.prepare(` INSERT INTO domain_routing (domain, prefer_playwright, http_failures, prefer_tls_impersonation, tls_success_count, last_updated) VALUES (?, 0, 0, ?, 1, datetime('now')) ON CONFLICT(domain) DO UPDATE SET tls_success_count = tls_success_count + 1, last_updated = datetime('now'), prefer_tls_impersonation = CASE WHEN tls_success_count + 1 >= ? THEN 1 ELSE prefer_tls_impersonation END `).run(domain, preferOnInsert, threshold); return getDomainRouting(domain); } ``` The router invokes this function after successful TLS fetches: ```ts // src/fetch/router.ts recordSuccess(domain) { try { recordTlsImpersonationSuccess(domain, getConfig().tlsSuccessThreshold); } catch { /* best-effort */ } } ``` ### Storing Anti-Bot Clearances When the browser tier solves a challenge (such as Cloudflare), Wigolo caches the clearance cookie, user agent, and expiry to reuse for subsequent requests: ```ts // src/cache/store.ts export function recordDomainClearance(host: string, clearance: DomainClearance): void { const db = getDatabase(); db.prepare(` INSERT INTO domain_routing (domain, prefer_playwright, http_failures, cf_clearance, clearance_ua, clearance_tier, clearance_expires_at, last_updated) VALUES (?, 0, 0, ?, ?, ?, ?, datetime('now')) ON CONFLICT(domain) DO UPDATE SET cf_clearance = excluded.cf_clearance, clearance_ua = excluded.clearance_ua, clearance_tier = excluded.clearance_tier, clearance_expires_at = excluded.clearance_expires_at, last_updated = datetime('now') `).run(host, clearance.cookie, clearance.ua, clearance.tier, clearance.expiresAt); } ``` Retrieval hides sensitive cookie values but reports presence and expiry: ```ts // src/cache/store.ts export function getDomainClearance(host: string): DomainClearance | null { const row = db.prepare(` SELECT cf_clearance, clearance_ua, clearance_tier, clearance_expires_at FROM domain_routing WHERE domain = ? LIMIT 1`).get(host); if (!row || row.cf_clearance == null) return null; return { cookie: row.cf_clearance, ua: row.clearance_ua ?? '', tier: row.clearance_tier ?? '', expiresAt: row.clearance_expires_at ?? '', }; } ``` ### Managing Backoff After Blocks Repeated 403 responses trigger a cool-down period recorded via `recordBackoff()`: ```ts // src/cache/store.ts export function recordBackoff(host: string, untilEpochMs: number): void { const db = getDatabase(); db.prepare(` INSERT INTO domain_routing (domain, prefer_playwright, http_failures, backoff_until, last_403_at, last_updated) VALUES (?, 0, 0, ?, datetime('now'), datetime('now')) ON CONFLICT(domain) DO UPDATE SET backoff_until = excluded.backoff_until, last_403_at = datetime('now'), last_updated = datetime('now') `).run(host, String(untilEpochMs)); } ``` The fetch ladder checks `getBackoff(host)` before attempting requests to respect polite delays. ## Inspecting Learned Routing with `wigolo tune` The `wigolo tune` command provides read-only access to the learning database through [`src/cli/tune.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cli/tune.ts). The CLI intentionally omits raw cookie values and user agents from output for security, showing only whether clearances are present. ### Listing All Domains ```bash wigolo tune list ``` This executes `listDomainRouting()` in [`src/cache/store.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cache/store.ts), which returns a summary projection: ```ts // src/cache/store.ts export function listDomainRouting(): DomainRoutingSummary[] { const rows = db.prepare(` SELECT domain, prefer_playwright, prefer_tls_impersonation, tls_success_count, http_failures, backoff_until, last_403_at, cf_clearance, clearance_expires_at FROM domain_routing ORDER BY domain`).all(); return rows.map(row => ({ domain: row.domain, preferBrowser: (row.prefer_playwright ?? 0) === 1, preferTlsImpersonation: (row.prefer_tls_impersonation ?? 0) === 1, tlsSuccessCount: row.tls_success_count ?? 0, httpFailures: row.http_failures ?? 0, backoffUntil: row.backoff_until ?? undefined, last403At: row.last_403_at ?? undefined, clearancePresent: row.cf_clearance != null, clearanceExpiresAt: row.clearance_expires_at ?? undefined, })); } ``` ### Viewing Specific Domain Details ```bash wigolo tune show example.com --json ``` Sample output showing learned preferences: ```json { "domain": "example.com", "preferBrowser": false, "preferTlsImpersonation": true, "tlsSuccessCount": 3, "httpFailures": 0, "clearancePresent": true, "clearanceExpiresAt": "2026-12-01 12:34:56" } ``` ## Clearing or Resetting Domain Learning When domains change their protection mechanisms or you need to troubleshoot fetch failures, you can reset the learned state without deleting the database. ### Reset Single Domain ```bash wigolo tune reset example.com ``` This calls `resetDomainRouting()` in [`src/cache/store.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cache/store.ts): ```ts // src/cache/store.ts export function resetDomainRouting(host: string): number { return db.prepare(`UPDATE domain_routing SET ${RESET_ROUTING_COLUMNS} WHERE domain = ?`) .run(host).changes; } ``` ### Reset All Domains ```bash wigolo tune reset --all ``` This invokes `resetAllDomainRouting()`: ```ts // src/cache/store.ts export function resetAllDomainRouting(): number { return db.prepare(`UPDATE domain_routing SET ${RESET_ROUTING_COLUMNS}`).run().changes; } ``` Both functions preserve the domain row in the table but nullify all learned columns, allowing fresh learning to begin immediately. ## Summary Wigolo's per-domain fetch learning system automatically optimizes request strategies through persistent SQLite storage: - **TLS tracking** increments `tls_success_count` and sets `prefer_tls_impersonation` when domains consistently accept TLS impersonation - **Clearance caching** stores solved anti-bot cookies with expiry timestamps to minimize repeated browser automation - **Backoff management** records cool-down periods after 403 responses to prevent rapid retry loops - **Inspection tools** via `wigolo tune list` and `show` reveal current preferences without exposing sensitive cookie values - **Reset capabilities** allow clearing learned state per-domain or globally using `wigolo tune reset` ## Frequently Asked Questions ### How do I check if Wigolo has learned to use TLS impersonation for a specific domain? Run `wigolo tune show --json` and check the `preferTlsImpersonation` field. If `true`, the fetch router in [`src/fetch/router.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/fetch/router.ts) will prefer the TLS tier over browser automation for that domain based on previous successful fetches recorded in `tls_success_count`. ### Where is the per-domain learning data physically stored? The data resides in a SQLite database at `/wigolo.db`, defaulting to `~/.wigolo/wigolo.db`. The `domain_routing` table defined in [`src/cache/migrations/008-antibot-clearance.sql`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cache/migrations/008-antibot-clearance.sql) contains all counters, preferences, and clearance cookies managed by [`src/cache/store.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cache/store.ts). ### Why doesn't `wigolo tune show` display the actual clearance cookie? The `listDomainRouting()` and related functions intentionally exclude raw `cf_clearance` values and user agents from CLI output to prevent accidental exposure of session tokens in terminal logs or shared environments. The JSON output only indicates `clearancePresent: true` with expiry timing. ### When should I reset domain learning data? Reset specific domains using `wigolo tune reset ` when sites change their protection mechanisms, when you suspect stale clearance cookies are causing blocks, or when debugging fetch failures after TLS or anti-bot updates. Use `wigolo tune reset --all` only when you want to force fresh learning across all previously visited domains.