# How News Subscriptions and Automated Research Digests Work in Local Deep Research

> Learn how Local Deep Research leverages automated news subscriptions to run scheduled searches, store results, and deliver curated findings in a unified digest feed.

- Repository: [learningcircuit/local-deep-research](https://github.com/learningcircuit/local-deep-research)
- Tags: how-to-guide
- Published: 2026-03-05

---

**Local Deep Research automatically runs scheduled searches through configurable news subscriptions, stores results as research history, and surfaces curated findings in a unified digest feed.**

Local Deep Research provides a comprehensive **news subscription** system that transforms static search queries into self-maintaining research pipelines. This open-source framework (`learningcircuit/local-deep-research`) enables automated monitoring of topics through periodic search execution against configured engines like SerpAPI or DuckDuckGo. Results aggregate into a personalized **automated research digest** accessible via REST API endpoints.

## Core Architecture Components

The system implements a four-layer architecture that separates web interface concerns from background processing logic.

### API Layer

The Flask routes in [`src/local_deep_research/web/routes/news_routes.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/web/routes/news_routes.py) expose RESTful endpoints for subscription management. Key routes include `POST /api/news/subscriptions` for creation, `GET /api/news/subscriptions` for listing, `GET /api/news/feed` for digest retrieval, and `POST /api/news/feedback` for voting. These endpoints handle authentication and delegate to the business logic layer.

### Business Logic Layer

Located in [`src/local_deep_research/news/api.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/api.py), this layer implements `create_subscription`, `update_subscription`, `delete_subscription`, `get_news_feed`, and `get_subscription_history`. The `create_subscription` function builds `NewsSubscription` ORM objects and explicitly notifies the scheduler via `_notify_scheduler_about_subscription_change("created")` to trigger immediate pickup.

### Subscription Model Layer

The abstract base class `BaseSubscription` in [`src/local_deep_research/news/subscription_manager/base_subscription.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/base_subscription.py) defines the contract for all subscriptions. Concrete implementations `SearchSubscription` ([`search_subscription.py`](https://github.com/learningcircuit/local-deep-research/blob/main/search_subscription.py)) and `TopicSubscription` ([`topic_subscription.py`](https://github.com/learningcircuit/local-deep-research/blob/main/topic_subscription.py)) implement `generate_search_query()` to produce query strings like `"site:nytimes.com {topic}"`. The model tracks `next_refresh`, `refresh_interval_minutes`, `is_active` state, and provides lifecycle hooks: `on_refresh_start()`, `on_refresh_success()`, and `on_refresh_error()`.

### Scheduler and Storage Layer

`NewsScheduler` in [`src/local_deep_research/news/subscription_manager/scheduler.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/scheduler.py) runs as a background worker, executing `run_once()` on a timer (typically every minute). It queries `SQLSubscriptionStorage` ([`storage.py`](https://github.com/learningcircuit/local-deep-research/blob/main/storage.py)) to load active subscriptions via methods like `list_by_user`. The storage layer handles `create`, `update_refresh_time`, `increment_stats`, and `delete` operations against SQLite or PostgreSQL backends.

## End-to-End Subscription Workflow

The platform converts a "search every 4 hours" definition into a continuous research pipeline through six distinct stages:

1. **Subscription Creation**: The Flask route invokes `news_api.create_subscription`, which persists a `NewsSubscription` record (defined in [`src/local_deep_research/database/models/news.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/database/models/news.py)) via `SQLSubscriptionStorage` and notifies the scheduler.

2. **Scheduler Detection**: `NewsScheduler.run_once()` loads all active subscriptions where `is_active` is true. For each subscription, `subscription.should_refresh()` evaluates whether the current UTC time exceeds the stored `next_refresh` timestamp.

3. **Query Generation and Execution**: Upon refresh trigger, the concrete subscription class executes `generate_search_query()`. The scheduler forwards this query to the generic search-engine factory (`src/local_deep_research/web_search_engines/...`), executing against configured backends. Results transform into `ResearchHistory` entries stored via `get_user_db_session(user_id)`.

4. **Post-Refresh Lifecycle Handling**: Success invokes `subscription.on_refresh_success(results)`, incrementing `refresh_count` via `increment_stats` and recalculating `next_refresh`. Failure triggers `subscription.on_refresh_error(exc)`, implementing exponential back-off and disabling the subscription after 10 consecutive failures.

5. **Digest Generation**: The `GET /api/news/feed` endpoint calls `news_api.get_news_feed`, filtering `ResearchHistory` (exposed in [`src/local_deep_research/database/models/__init__.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/database/models/__init__.py)) for entries with news metadata (`generated_headline`, `is_news_search`). The function returns up to 20 news cards containing headlines, summaries, and source links.

6. **User Interaction and Feedback**: Users vote on individual news cards via `POST /api/news/feedback`, which updates ratings in the `UserRating` table. Aggregated votes are retrieved via `get_votes_for_cards` to influence future rankings.

## Working with the News API

### Creating a Subscription

```python
import requests

payload = {
    "query": "quantum computing breakthroughs",
    "type": "search",
    "refresh_minutes": 180,
    "name": "Quantum Updates",
    "is_active": True,
    "search_engine": "serper",
}
resp = requests.post(
    "https://your-instance/api/news/subscriptions",
    json=payload,
    cookies={"session": "..."}
)
print(resp.json())

```

### Retrieving the Automated Digest

```python
resp = requests.get(
    "https://your-instance/api/news/feed?limit=10",
    cookies={"session": "..."}
)
feed = resp.json()["news_items"]
for card in feed:
    print(f"{card['headline']} – {card['summary']}")

```

### Updating Subscription Parameters

```python
update_payload = {"refresh_minutes": 240}
resp = requests.patch(
    "https://your-instance/api/news/subscriptions/abcd-1234",
    json=update_payload,
    cookies={"session": "..."}
)

```

## Background Scheduler Implementation

The `NewsScheduler` orchestrates automated refreshes without user interaction.

```python
from local_deep_research.news.subscription_manager.scheduler import get_news_scheduler

scheduler = get_news_scheduler()
while True:
    scheduler.run_once()
    time.sleep(60)

```

Internally, `run_once()` iterates through active subscriptions, validates refresh timing with `should_refresh()`, executes searches, and persists history records to the database.

## Key Implementation Files

- **[`src/local_deep_research/web/routes/news_routes.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/web/routes/news_routes.py)**: Flask endpoints for subscription CRUD, feed access, and feedback submission.
- **[`src/local_deep_research/news/api.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/api.py)**: Core business logic including `create_subscription`, `get_news_feed`, and `update_subscription`.
- **[`src/local_deep_research/news/subscription_manager/base_subscription.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/base_subscription.py)**: Abstract `BaseSubscription` class with timing logic and error handling hooks.
- **[`src/local_deep_research/news/subscription_manager/search_subscription.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/search_subscription.py)**: Concrete implementation for free-text search subscriptions.
- **[`src/local_deep_research/news/subscription_manager/topic_subscription.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/topic_subscription.py)**: Concrete implementation for topic-based subscriptions.
- **[`src/local_deep_research/news/subscription_manager/scheduler.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/scheduler.py)**: Background worker implementing `NewsScheduler` and `run_once()`.
- **[`src/local_deep_research/news/subscription_manager/storage.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/news/subscription_manager/storage.py)**: `SQLSubscriptionStorage` class handling persistence and statistics.
- **[`src/local_deep_research/database/models/news.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/database/models/news.py)**: ORM definitions for `NewsSubscription` and `UserRating` tables.
- **[`src/local_deep_research/database/models/__init__.py`](https://github.com/learningcircuit/local-deep-research/blob/main/src/local_deep_research/database/models/__init__.py)**: Exposes `ResearchHistory` used by the feed generator.

## Summary

- **News subscriptions** in Local Deep Research are persistent search definitions that execute on configurable intervals through `BaseSubscription` subclasses like `SearchSubscription` and `TopicSubscription`.
- The **`NewsScheduler`** background worker checks `should_refresh()` every minute, executing queries through the generic search-engine wrapper and storing results as `ResearchHistory` entries.
- **Automated research digests** surface via `GET /api/news/feed`, which filters history for news-specific metadata and returns curated cards with headlines, summaries, and voting capabilities.
- The system implements robust error handling with exponential back-off, disabling subscriptions after 10 consecutive failures via `on_refresh_error()`.
- User interactions including pause/resume operations and card voting persist through `SQLSubscriptionStorage` and the `UserRating` table.

## Frequently Asked Questions

### How does the scheduler know when to run a subscription?

The `NewsScheduler` loads all active subscriptions from the database every minute and calls `subscription.should_refresh()`, which compares the current UTC timestamp against the `next_refresh` field stored in the subscription record.

### What happens if a scheduled search fails?

The system invokes `subscription.on_refresh_error(exc)`, which increments an internal error counter using `increment_stats` and applies exponential back-off to the next refresh time. After 10 consecutive failures, the subscription automatically sets `is_active` to false to prevent further resource consumption.

### Can I use different search engines for different subscriptions?

Yes. Each subscription stores a `search_engine` parameter (e.g., `"serper"`, `"duckduckgo"`). When the scheduler executes `generate_search_query()`, it passes the query to the search-engine factory configured for that specific subscription type, allowing per-subscription engine selection.

### How does the research digest differ from regular search history?

While both store results in `ResearchHistory`, the news feed specifically filters for entries containing news metadata flags like `generated_headline` and `is_news_search`. The `get_news_feed` function transforms these into structured cards with voting capabilities via the `UserRating` table, whereas standard search history appears in the general research log without digest formatting.