Where Missing Markets Are Stored in Poly Data: Complete File Location Guide

Missing markets in Poly Data are stored in a CSV file named missing_markets.csv in the repository root, with the get_markets and update_missing_tokens functions in poly_utils/utils.py handling all read and write operations.

If you're working with the warproxxx/poly_data repository and need to locate or manage markets that couldn't be fetched during data collection, this guide covers the exact file paths, function signatures, and usage patterns you need.

The Default Missing Markets File Location

Poly Data uses a simple, predictable file structure for handling incomplete market data. When the update_missing_tokens function discovers new token IDs that need to be fetched, it persists them to a dedicated CSV file.

Primary Storage: missing_markets.csv

The default location where missing markets are stored is:

missing_markets.csv

This file resides in the working directory—typically the repository root—unless you specify an alternative path when calling the utility functions.

Core Functions in poly_utils/utils.py

All operations related to where missing markets are stored are handled by two functions in poly_utils/utils.py:

get_markets: Loading Markets Including Missing Ones

The get_markets function automatically checks for and loads the missing markets file if it exists:

def get_markets(main_file: str = "markets.csv",
                missing_file: str = "missing_markets.csv"):
    # …

    # Load missing markets file

    if os.path.exists(missing_file):
        missing_df = pl.scan_csv(missing_file, …).collect(streaming=True)
        dfs.append(missing_df)
        print(f"Loaded {len(missing_df)} markets from {missing_file}")

This design ensures that missing markets are seamlessly integrated with regular market data when loaded.

update_missing_tokens: Writing Missing Markets to File

When new missing token IDs are discovered, update_missing_tokens fetches their data and writes (or overwrites) the missing markets file:

def update_missing_tokens(missing_token_ids: List[str],
                          csv_filename: str = "missing_markets.csv"):
    # …

    # Save the dataframe to the CSV file

    df.write_csv(csv_filename)
    print(f"Saved missing markets to {csv_filename}")

Practical Code Examples

Loading All Markets Including Missing Ones

# Example: Load all markets, including any missing ones

from poly_utils.utils import get_markets

# Returns a Polars DataFrame with both regular and missing markets

markets_df = get_markets()          # uses defaults: markets.csv + missing_markets.csv

print(markets_df.head())

Updating the Missing Markets File

# Example: Update the missing‑markets file after discovering new token IDs

from poly_utils.utils import update_missing_tokens

new_missing_ids = ["0x1234…", "0xabcd…"]
update_missing_tokens(new_missing_ids)   # writes to missing_markets.csv

Customizing Where Missing Markets Are Stored

Both functions accept optional parameters to change where missing markets are stored:

Parameter	Default	Purpose
missing_file in get_markets	"missing_markets.csv"	Path to read missing markets from
csv_filename in update_missing_tokens	"missing_markets.csv"	Path to write missing markets to

Example with custom path:

from poly_utils.utils import get_markets, update_missing_tokens

# Use a subdirectory for missing markets

markets_df = get_markets(missing_file="data/missing_markets.csv")

# Write to the same custom location

update_missing_tokens(new_ids, csv_filename="data/missing_markets.csv")

Summary

• Missing markets are stored in missing_markets.csv by default, located in the working directory
• The get_markets function in poly_utils/utils.py automatically loads this file when present
• The update_missing_tokens function writes new missing market data to the same file
• Both functions support custom file paths via the missing_file and csv_filename parameters
• The implementation uses Polars for efficient CSV reading and writing with streaming support

Frequently Asked Questions

What happens if missing_markets.csv doesn't exist?

If the file doesn't exist, get_markets simply returns the regular markets from markets.csv without error. The missing markets file is only loaded when os.path.exists(missing_file) returns True.

Can I use a different filename for missing markets?

Yes. Pass your preferred filename to both functions: use missing_file in get_markets() and csv_filename in update_missing_tokens(). Both parameters accept relative or absolute paths.

Does update_missing_tokens append or overwrite existing data?

The function overwrites the entire file. When you call df.write_csv(csv_filename), it replaces any existing content. If you need to preserve existing missing markets, load them first with get_markets and merge before calling update_missing_tokens.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}