# Setting Up the Watchlist for Automation with last30days-skill > Automate recurring research tasks with the last30days-skill watchlist. Store topics in SQLite, run scheduled tasks, and manage token budgets efficiently using watchlist.py CLI. - Repository: [Matt Van Horn/last30days-skill](https://github.com/mvanhorn/last30days-skill) - Tags: how-to-guide - Published: 2026-03-25 --- **The last30days-skill watchlist system lets you automate recurring research tasks by storing topics in SQLite, running them on schedules, and respecting daily token budgets via the [`watchlist.py`](https://github.com/mvanhorn/last30days-skill/blob/main/watchlist.py) CLI.** The **last30days-skill** repository (`mvanhorn/last30days-skill`) is a Python-based research engine that aggregates community signals from Reddit, X, YouTube, and other platforms into concise briefings. For ongoing investigations, the integrated **watchlist** subsystem automates periodic research runs while tracking costs and deduplicating results in a local database. ## Understanding the Watchlist Architecture The watchlist consists of three core components that work together to schedule, execute, and persist research findings. ### The Watchlist CLI ([`scripts/watchlist.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/watchlist.py)) The primary interface for automation resides in [`scripts/watchlist.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/watchlist.py), which implements `argparse` sub-commands including `add`, `remove`, `list`, `run-one`, `run-all`, and `config`. This script manages topic schedules stored in SQLite and launches the research engine via `subprocess.run` when deadlines trigger. ### SQLite Persistence ([`scripts/store.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/store.py)) All watchlist data lives in a local database at `~/.local/share/last30days/research.db`, initialized automatically on first use. The schema (v1) defines tables for `topics`, `research_runs`, `findings`, and `settings`, with the database opened in **WAL** mode for safe concurrent access. Full-text search is powered by an **FTS5** virtual table (`findings_fts`) using a Porter+Unicode tokenizer. ### The Research Engine ([`scripts/last30days.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/last30days.py)) When invoked by the watchlist runner, [`scripts/last30days.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/last30days.py) executes parallel source searches using `ThreadPoolExecutor`, aggregates results, and returns JSON data that [`watchlist.py`](https://github.com/mvanhorn/last30days-skill/blob/main/watchlist.py) normalizes and stores via `store.store_findings`. ## Initial Setup and Configuration Before automating, ensure your environment meets the requirements and configure necessary API keys. ### Prerequisites - **Python 3.10+** and **Node.js** (required for the Bird X client) - **Optional API keys**: `SCRAPECREATORS_API_KEY`, `XAI_API_KEY`, and platform-specific tokens for X or Bluesky - A writable home directory for the SQLite database ### Database Initialization The database and its schema are created automatically when you first run a [`watchlist.py`](https://github.com/mvanhorn/last30days-skill/blob/main/watchlist.py) command. No manual migration steps are required. ## Adding and Managing Watchlist Topics Topics represent persistent research queries that run on defined schedules. ### Adding a Topic Use the `add` sub-command to create a new entry with a cron-style schedule (default: `0 8 * * *` for 08:00 UTC daily): ```bash python3 scripts/watchlist.py add "AI video tools" --schedule "0 9 * * *" ``` To add custom search queries or weekly execution: ```bash python3 scripts/watchlist.py add "NVIDIA news" --weekly --queries "NVIDIA earnings,NVIDIA GPU launch" ``` The `add` function calls `store.add_topic()` to persist the topic, queries, and schedule metadata to the SQLite `topics` table. ### Listing and Removing Topics View all configured topics, their schedules, and finding counts: ```bash python3 scripts/watchlist.py list ``` Remove a topic by name to delete its record and associated schedule: ```bash python3 scripts/watchlist.py remove "AI video tools" ``` ## Automating Research Execution The watchlist supports both manual one-off runs and fully automated scheduling. ### Running a Single Topic Manually Test a specific topic or force an immediate update without waiting for the schedule: ```bash python3 scripts/watchlist.py run-one "AI video tools" ``` This spawns [`last30days.py`](https://github.com/mvanhorn/last30days-skill/blob/main/last30days.py) with `--emit=json`, captures the output, normalizes items, and persists them using `store.store_findings()`. ### Batch Execution with `run-all` Process all enabled topics that are due for execution and respect the daily budget: ```bash python3 scripts/watchlist.py run-all ``` Before executing, the runner checks `store.get_daily_cost()` against the configured `daily_budget`. If the cumulative token cost exceeds the limit, remaining topics are skipped and reported in the JSON payload. ### Scheduling with Cron For hands-off automation, add a cron entry that invokes `run-all` hourly: ```cron 0 * * * * /usr/bin/python3 /path/to/last30days-skill/scripts/watchlist.py run-all >> /tmp/last30days-watch.log 2>&1 ``` The internal schedule logic determines which specific topics trigger during each cron invocation based on their stored cron expressions. ## Configuring Budgets and Querying Results Control costs and inspect accumulated findings using the CLI and direct database queries. ### Setting a Daily Token Budget Prevent runaway API costs by setting a daily limit (in dollars): ```bash python3 scripts/watchlist.py config budget 10.00 ``` This updates the `settings` table in SQLite, and subsequent `run-all` executions enforce the limit via `get_daily_cost()`. ### Searching Stored Findings Query the full-text index across titles, summaries, and content: ```bash python3 scripts/store.py search "json prompts" ``` Retrieve recent findings for a specific topic (e.g., topic ID 3) within a time window: ```bash python3 scripts/store.py query --topic-id 3 --since 7d ``` Direct SQL access to `~/.local/share/last30days/research.db` is also available for custom analytics. ## Summary - **The watchlist system** in `mvanhorn/last30days-skill` automates recurring research through [`scripts/watchlist.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/watchlist.py), [`scripts/store.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/store.py), and [`scripts/last30days.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/last30days.py). - **Topics** are stored in SQLite with cron schedules, custom queries, and enabled/disabled flags. - **Daily budgets** are enforced by checking cumulative token costs against the `settings` table before each run. - **Automation** is typically configured via cron running `watchlist.py run-all`, which handles scheduling logic and deduplication automatically. - **Results** persist to `~/.local/share/last30days/research.db` with FTS5 indexing for fast full-text retrieval. ## Frequently Asked Questions ### Where is the watchlist database stored? The SQLite database is created automatically at `~/.local/share/last30days/research.db` on first use. It uses WAL mode for concurrent safety and contains tables for `topics`, `research_runs`, `findings`, and `settings`, along with an FTS5 virtual table for full-text search. ### How does the daily budget limit work? When `watchlist.py run-all` executes, it calls `store.get_daily_cost()` to sum the token costs from `research_runs` for the current day. If this total exceeds the value set via `watchlist.py config budget`, the runner skips remaining topics and notes the limit hit in the output JSON. ### Can I run a single watchlist topic without waiting for the schedule? Yes. Use `python3 scripts/watchlist.py run-one "Topic Name"` to execute the research engine immediately for a specific topic. This command bypasses the schedule check, runs the query against all configured sources, and stores the findings in the database. ### What sources does the automated research include? According to the source code in [`scripts/last30days.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/last30days.py), the engine searches Reddit, X (via the Bird client in [`scripts/lib/bird_x.py`](https://github.com/mvanhorn/last30days-skill/blob/main/scripts/lib/bird_x.py)), YouTube, TikTok, Instagram, Bluesky, Truth Social, Polymarket, and the open web (via Brave, OpenRouter, or parallel search adapters). Results are aggregated, scored, deduplicated, and stored automatically.