# How Does Poly Data Fetch Market Data from Polymarket?

> Discover how Poly Data fetches market data from Polymarket using Python routines for full crawls and targeted retrieval. Store and analyze normalized data efficiently.

- Repository: [warproxxx/poly_data](https://github.com/warproxxx/poly_data)
- Tags: how-to-guide
- Published: 2026-04-21

---

**Poly Data fetches market data from Polymarket's public Gamma API using two complementary Python routines—`update_markets()` for full chronological crawls and `update_missing_tokens()` for targeted retrieval by token ID—storing normalized results in CSV format for downstream Polars analysis.**

Poly Data is an open-source toolkit ([warproxxx/poly_data](https://github.com/warproxxx/poly_data)) designed to extract prediction market data from Polymarket. The system implements a robust pagination strategy that handles rate limits, parses JSON-encoded metadata, and maintains a stable local cache of market information in `markets.csv`.

## Full Market Crawl via `update_markets`

The primary ingestion routine is `update_markets()`, located in **[`update_utils/update_markets.py`](https://github.com/warproxxx/poly_data/blob/main/update_utils/update_markets.py)**. This function performs a chronological crawl of Polymarket’s Gamma API endpoint (`https://gamma-api.polymarket.com/markets`), downloading every market in creation order.

### Deterministic Pagination Strategy

The crawler implements resume capability by counting existing records before issuing requests. The helper `count_csv_lines()` determines the starting offset, allowing the script to append new data without duplication.

```python
current_offset = count_csv_lines(csv_filename)
mode = 'a' if file_exists else 'w'

```

API requests use four query parameters to ensure deterministic ordering:

```python
params = {
    'order': 'createdAt',
    'ascending': 'true',
    'limit': batch_size,
    'offset': current_offset
}
response = requests.get(base_url, params=params, timeout=30)

```

The script advances the offset by the actual number of markets written (not the requested batch size), stopping when a returned batch is smaller than `batch_size`—signaling exhaustion of the API dataset.

### Robust Retry Handling

The implementation includes specific backoff strategies for HTTP error codes:

- **HTTP 500**: Wait 5 seconds and retry
- **HTTP 429**: Wait 10 seconds and retry  
- **Other non-200**: Log error and retry after 3 seconds

This ensures reliable completion across large historical syncs without manual intervention.

### Parsing and CSV Storage

Market objects from the Gamma API contain JSON-encoded strings for nested fields. The parser explicitly decodes `outcomes` and `clobTokenIds` using `json.loads()` before extracting individual values:

```python
outcomes = json.loads(market.get('outcomes', '[]'))
clob_tokens = json.loads(market.get('clobTokenIds', '[]'))

```

Each row is written to the CSV with a fixed schema: `createdAt, id, question, outcome1, outcome2, token_id_1, token_id_2, negativeRisk, ticker, closedTime`. This standardized format allows downstream modules to consume data predictably.

## Targeted Retrieval via `update_missing_tokens`

When specific markets are missing from the primary crawl (e.g., markets added after initial sync or skipped records), **[`poly_utils/utils.py`](https://github.com/warproxxx/poly_data/blob/main/poly_utils/utils.py)** provides `update_missing_tokens()` to fill gaps individually.

### Token-Based Filtering

Unlike the full crawl which uses offset pagination, this routine queries the same Gamma API endpoint using the `clob_token_ids` parameter to retrieve specific markets by their CLOB token ID:

```python
response = requests.get(
    'https://gamma-api.polymarket.com/markets',
    params={'clob_token_ids': token_id},
    timeout=30
)

```

This approach is essential for fetching markets that were not captured in the chronological crawl due to API hiccups or late additions.

### Deduplication Logic

Before writing new records, the function reads existing IDs from the target CSV into a `processed_market_ids` set to prevent duplication:

```python
with open(csv_filename, 'r', encoding='utf-8') as f:
    reader = csv.DictReader(f)
    for row in reader:
        processed_market_ids.add(row['id'])

```

Records are only appended if their market ID is not already present, ensuring idempotent operations when run multiple times.

## Data Consumption with Polars

Once `markets.csv` (and optionally `missing_markets.csv`) is populated, **[`poly_utils/utils.py`](https://github.com/warproxxx/poly_data/blob/main/poly_utils/utils.py)** exposes `get_markets()` to load the data into a Polars DataFrame:

```python
from poly_utils.utils import get_markets

df = get_markets()  # Returns Polars DataFrame sorted by createdAt

```

This function reads both CSV files, deduplicates on the market `id` column, and returns a unified `combined_df`. The DataFrame is used throughout the repository—for example, [`process_live.py`](https://github.com/warproxxx/poly_data/blob/main/process_live.py) converts this wide format into a long format for time-series analysis.

## Practical Implementation Examples

### Running a Full Market Update

To fetch all markets in batches of 500 (default) and store them in `markets.csv`:

```python
from update_utils.update_markets import update_markets

update_markets()

```

Alternatively, run the convenience wrapper:

```python
from update_all import main
main()  # Executes update_markets() and handles CSV initialization

```

### Filling Specific Market Gaps

To retrieve markets by their CLOB token IDs:

```python
from poly_utils.utils import update_missing_tokens

missing_ids = [
    "0x1234567890abcdef...",
    "0xfedcba0987654321..."
]
update_missing_tokens(missing_ids)

```

### Loading Data for Analysis

To access the consolidated market table in your analysis pipeline:

```python
from poly_utils.utils import get_markets

markets_df = get_markets()
print(markets_df.head())

```

## Summary

- **Poly Data** fetches market data from the Polymarket Gamma API endpoint (`https://gamma-api.polymarket.com/markets`).
- **`update_markets()`** in [`update_utils/update_markets.py`](https://github.com/warproxxx/poly_data/blob/main/update_utils/update_markets.py) performs chronological pagination using `createdAt` ordering, with robust retry logic for HTTP 500/429 errors.
- **`update_missing_tokens()`** in [`poly_utils/utils.py`](https://github.com/warproxxx/poly_data/blob/main/poly_utils/utils.py) enables targeted fetching by `clob_token_ids` and implements CSV-level deduplication.
- Both routines output to a standardized CSV schema consumed by **`get_markets()`**, which returns a Polars DataFrame for downstream analysis.
- The system supports resume capability via line-counting offsets, making it suitable for incremental updates of large datasets.

## Frequently Asked Questions

### What API endpoint does Poly Data use to fetch market data?

Poly Data queries Polymarket's public **Gamma API** at `https://gamma-api.polymarket.com/markets`. The full crawl uses query parameters for pagination (`order=createdAt`, `ascending=true`, `limit`, `offset`), while targeted fetches use the `clob_token_ids` filter parameter.

### How does Poly Data handle API rate limits and errors?

The codebase implements specific retry logic with exponential backoff. When the API returns **HTTP 429** (rate limit), the script waits 10 seconds before retrying. For **HTTP 500** errors, it waits 5 seconds. All other non-200 responses trigger a 3-second wait and retry cycle, ensuring robust completion across unstable network conditions.

### What is the difference between `update_markets` and `update_missing_tokens`?

**`update_markets()`** performs a full chronological crawl of all markets using offset pagination, storing every record in `markets.csv`. **`update_missing_tokens()`** is designed for targeted retrieval—accepting a list of specific CLOB token IDs, querying the API for just those markets, and appending them to the CSV only if not already present.

### How is the fetched market data stored and accessed?

Data is persisted to CSV files (primarily `markets.csv`) with a fixed column schema including `createdAt`, `id`, `question`, outcome strings, token IDs, and metadata. The **`get_markets()`** function in [`poly_utils/utils.py`](https://github.com/warproxxx/poly_data/blob/main/poly_utils/utils.py) reads these CSVs into a Polars DataFrame, deduplicates by market ID, and returns the data sorted by creation timestamp for analysis.