Setting Up the Watchlist for Automation with last30days-skill

The last30days-skill watchlist system lets you automate recurring research tasks by storing topics in SQLite, running them on schedules, and respecting daily token budgets via the watchlist.py CLI.

The last30days-skill repository (mvanhorn/last30days-skill) is a Python-based research engine that aggregates community signals from Reddit, X, YouTube, and other platforms into concise briefings. For ongoing investigations, the integrated watchlist subsystem automates periodic research runs while tracking costs and deduplicating results in a local database.

Understanding the Watchlist Architecture

The watchlist consists of three core components that work together to schedule, execute, and persist research findings.

The Watchlist CLI (scripts/watchlist.py)

The primary interface for automation resides in scripts/watchlist.py, which implements argparse sub-commands including add, remove, list, run-one, run-all, and config. This script manages topic schedules stored in SQLite and launches the research engine via subprocess.run when deadlines trigger.

SQLite Persistence (scripts/store.py)

All watchlist data lives in a local database at ~/.local/share/last30days/research.db, initialized automatically on first use. The schema (v1) defines tables for topics, research_runs, findings, and settings, with the database opened in WAL mode for safe concurrent access. Full-text search is powered by an FTS5 virtual table (findings_fts) using a Porter+Unicode tokenizer.

The Research Engine (scripts/last30days.py)

When invoked by the watchlist runner, scripts/last30days.py executes parallel source searches using ThreadPoolExecutor, aggregates results, and returns JSON data that watchlist.py normalizes and stores via store.store_findings.

Initial Setup and Configuration

Before automating, ensure your environment meets the requirements and configure necessary API keys.

Prerequisites

• Python 3.10+ and Node.js (required for the Bird X client)
• Optional API keys: SCRAPECREATORS_API_KEY, XAI_API_KEY, and platform-specific tokens for X or Bluesky
• A writable home directory for the SQLite database

Database Initialization

The database and its schema are created automatically when you first run a watchlist.py command. No manual migration steps are required.

Adding and Managing Watchlist Topics

Topics represent persistent research queries that run on defined schedules.

Adding a Topic

Use the add sub-command to create a new entry with a cron-style schedule (default: 0 8 * * * for 08:00 UTC daily):

python3 scripts/watchlist.py add "AI video tools" --schedule "0 9 * * *"

To add custom search queries or weekly execution:

python3 scripts/watchlist.py add "NVIDIA news" --weekly --queries "NVIDIA earnings,NVIDIA GPU launch"

The add function calls store.add_topic() to persist the topic, queries, and schedule metadata to the SQLite topics table.

Listing and Removing Topics

View all configured topics, their schedules, and finding counts:

python3 scripts/watchlist.py list

Remove a topic by name to delete its record and associated schedule:

python3 scripts/watchlist.py remove "AI video tools"

Automating Research Execution

The watchlist supports both manual one-off runs and fully automated scheduling.

Running a Single Topic Manually

Test a specific topic or force an immediate update without waiting for the schedule:

python3 scripts/watchlist.py run-one "AI video tools"

This spawns last30days.py with --emit=json, captures the output, normalizes items, and persists them using store.store_findings().

Batch Execution with run-all

Process all enabled topics that are due for execution and respect the daily budget:

python3 scripts/watchlist.py run-all

Before executing, the runner checks store.get_daily_cost() against the configured daily_budget. If the cumulative token cost exceeds the limit, remaining topics are skipped and reported in the JSON payload.

Scheduling with Cron

For hands-off automation, add a cron entry that invokes run-all hourly:

0 * * * * /usr/bin/python3 /path/to/last30days-skill/scripts/watchlist.py run-all >> /tmp/last30days-watch.log 2>&1

The internal schedule logic determines which specific topics trigger during each cron invocation based on their stored cron expressions.

Configuring Budgets and Querying Results

Control costs and inspect accumulated findings using the CLI and direct database queries.

Setting a Daily Token Budget

Prevent runaway API costs by setting a daily limit (in dollars):

python3 scripts/watchlist.py config budget 10.00

This updates the settings table in SQLite, and subsequent run-all executions enforce the limit via get_daily_cost().

Searching Stored Findings

Query the full-text index across titles, summaries, and content:

python3 scripts/store.py search "json prompts"

Retrieve recent findings for a specific topic (e.g., topic ID 3) within a time window:

python3 scripts/store.py query --topic-id 3 --since 7d

Direct SQL access to ~/.local/share/last30days/research.db is also available for custom analytics.

Summary

• The watchlist system in mvanhorn/last30days-skill automates recurring research through scripts/watchlist.py, scripts/store.py, and scripts/last30days.py.
• Topics are stored in SQLite with cron schedules, custom queries, and enabled/disabled flags.
• Daily budgets are enforced by checking cumulative token costs against the settings table before each run.
• Automation is typically configured via cron running watchlist.py run-all, which handles scheduling logic and deduplication automatically.
• Results persist to ~/.local/share/last30days/research.db with FTS5 indexing for fast full-text retrieval.

Frequently Asked Questions

Where is the watchlist database stored?

The SQLite database is created automatically at ~/.local/share/last30days/research.db on first use. It uses WAL mode for concurrent safety and contains tables for topics, research_runs, findings, and settings, along with an FTS5 virtual table for full-text search.

How does the daily budget limit work?

When watchlist.py run-all executes, it calls store.get_daily_cost() to sum the token costs from research_runs for the current day. If this total exceeds the value set via watchlist.py config budget, the runner skips remaining topics and notes the limit hit in the output JSON.

Can I run a single watchlist topic without waiting for the schedule?

Yes. Use python3 scripts/watchlist.py run-one "Topic Name" to execute the research engine immediately for a specific topic. This command bypasses the schedule check, runs the query against all configured sources, and stores the findings in the database.

What sources does the automated research include?

According to the source code in scripts/last30days.py, the engine searches Reddit, X (via the Bird client in scripts/lib/bird_x.py), YouTube, TikTok, Instagram, Bluesky, Truth Social, Polymarket, and the open web (via Brave, OpenRouter, or parallel search adapters). Results are aggregated, scored, deduplicated, and stored automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}