# How dataforseo.py Retrieves SERP Positions and Keyword Metrics for Competitive Analysis

> Discover how dataforseo.py retrieves SERP positions and keyword metrics from DataForSEO API. Analyze competitors, search volume, and CPC effectively.

- Repository: [Craig/seomachine](https://github.com/TheCraigHewitt/seomachine)
- Tags: how-to-guide
- Published: 2026-03-12

---

**The [`dataforseo.py`](https://github.com/TheCraigHewitt/seomachine/blob/main/dataforseo.py) module acts as a thin Python client for the DataForSEO API, using environment-based Basic Auth to POST keyword tasks to `/v3/serp/google/organic/live/advanced`, then parsing the JSON response to extract exact SERP positions, search volume, CPC, and competitor gaps.**

The [`dataforseo.py`](https://github.com/TheCraigHewitt/seomachine/blob/main/dataforseo.py) file, located in `data_sources/modules/` within TheCraigHewitt/seomachine, provides the core integration with the DataForSEO API. This lightweight client enables the SEO Machine platform to retrieve live SERP positions and comprehensive keyword metrics necessary for data-driven competitive analysis.

## Authentication and Session Management in dataforseo.py

### Environment-Based Credentials

The `DataForSEO` class initializes by reading `DATAFORSEO_LOGIN` and `DATAFORSEO_PASSWORD` from environment variables (lines 24-36). These credentials are encoded into a **Basic Auth** header using Base64, which the class attaches to a reusable `requests.Session` object stored as `self.session`.

### The _post() Helper Method

All API calls flow through the private `_post()` method (lines 42-48). This helper accepts an endpoint path and a JSON payload, constructs the full URL by appending the path to the base URL (`https://api.dataforseo.com`), and returns the parsed JSON response. Using a single session ensures connection pooling and automatic header injection for every request.

## Retrieving SERP Positions with get_rankings()

The `get_rankings()` method (lines 49-100) determines exactly where a target domain appears in Google’s organic results for a list of keywords.

### Task Construction and API Endpoint

For each keyword provided, the method builds a task dictionary containing `keyword`, `location_code`, `language_code`, `device`, and `os` (lines 60-68). It then POSTs the entire task list to the **`/v3/serp/google/organic/live/advanced`** endpoint (line 78).

### Parsing Positions and Metrics

Upon receiving the response, `get_rankings()` iterates through the `tasks` array. For each successful task (status code `20000`), it extracts the `items` array holding organic results (lines 83-87). The method enumerates these items starting at 1, checking each `domain` field against the target domain. The first match records the **position** and **URL** (lines 88-95).

Finally, the method enriches each result with **search volume** and **CPC** data pulled from `keyword_data.keyword_info` (lines 99-104), returning a list of dictionaries containing `keyword`, `domain`, `position`, `url`, `ranking` (boolean), `search_volume`, and `cpc`.

```python
from data_sources.modules.dataforseo import DataForSEO

dfs = DataForSEO()                     # reads credentials from .env

rankings = dfs.get_rankings(
    domain="example.com",
    keywords=["seo tools", "keyword research", "content optimization"]
)

for r in rankings:
    print(f"{r['keyword']}: position={r['position'] or '–'}, volume={r['search_volume']}")

```

## Extracting Full Keyword Metrics via get_serp_data()

While `get_rankings()` focuses on position tracking, `get_serp_data()` (lines 108-174) retrieves a comprehensive snapshot of a single keyword’s SERP landscape.

### Organic Results and SERP Features

The method constructs a single-keyword task with a configurable `depth` parameter (lines 125-132) and POSTs to the same organic endpoint. After validating the response (lines 136-142), it parses:
- **`organic_results`**: A list of dictionaries containing `position`, `url`, `domain`, `title`, `description`, and `breadcrumb` (lines 146-157).
- **`features`**: Non-organic SERP elements such as “carousel” or “knowledge_graph” collected from the result items (lines 158-163).

### Keyword-Level Intelligence

Beyond the SERP structure, the method extracts quantitative metrics from `keyword_data.keyword_info`: **search_volume**, **cpc**, and **competition** (lines 164-170). These values enable the SEO Machine to calculate opportunity scores and prioritize keywords based on traffic potential and cost-per-click data.

```python
serp = dfs.get_serp_data("seo automation", limit=50)

print("Search volume:", serp["search_volume"])
print("CPC:", serp["cpc"])
print("Top organic result:", serp["organic_results"][0]["url"])
print("SERP features:", ", ".join(serp["features"]))

```

## Competitive Gap Analysis with analyze_competitor()

The `analyze_competitor()` method (lines 176-233) leverages the previous methods to perform side-by-side ranking comparisons.

### Position Gap Calculation

For each keyword in the analysis set, the method retrieves SERP data and records the position of both the **competitor domain** and **your domain** (lines 213-219). It calculates the **gap** as `your_position - competitor_position`, where negative values indicate the competitor outranks you.

### Opportunity Scoring

Based on the gap size and whether you rank at all, the method assigns an **opportunity level** (high, medium, low) to each keyword (line 231). This scoring system allows SEO teams to prioritize keywords where competitors have significant advantages but search volume justifies the effort to close the gap.

```python
comparison = dfs.analyze_competitor(
    competitor_domain="competitor.com",
    keywords=["seo audit", "link building", "technical seo"],
    your_domain="example.com"
)

for entry in comparison["comparison"]:
    print(f"{entry['keyword']}: competitor={entry['competitor_position']}, "
          f"you={entry['your_position']}, gap={entry['gap']}, "
          f"opportunity={entry['opportunity']}")

```

## Summary

- **[`dataforseo.py`](https://github.com/TheCraigHewitt/seomachine/blob/main/dataforseo.py)** acts as a thin Python client for the DataForSEO API, handling authentication via environment variables and a reusable `requests.Session`.
- The **`get_rankings()`** method retrieves exact SERP positions for a target domain across multiple keywords, enriching results with search volume and CPC data.
- **`get_serp_data()`** provides comprehensive keyword intelligence, including organic results, SERP features, and competitive metrics like competition level.
- **`analyze_competitor()`** compares your rankings against competitors, calculating position gaps and assigning opportunity scores to prioritize SEO efforts.

## Frequently Asked Questions

### How does dataforseo.py handle API authentication?

The `DataForSEO` class reads `DATAFORSEO_LOGIN` and `DATAFORSEO_PASSWORD` from environment variables during initialization. It encodes these credentials into a Basic Auth header using Base64 and attaches them to a reusable `requests.Session` object, ensuring every subsequent API request includes proper authentication automatically.

### What SERP data does the get_rankings() method return?

The `get_rankings()` method returns a list of dictionaries, each containing the target `keyword`, `domain`, `position` (the exact SERP index where the domain appears), the ranking `url`, a boolean `ranking` flag, plus keyword metrics including `search_volume` and `cpc` pulled from the DataForSEO API's keyword_info object.

### Can dataforseo.py analyze multiple competitors simultaneously?

The current implementation of `analyze_competitor()` in [`data_sources/modules/dataforseo.py`](https://github.com/TheCraigHewitt/seomachine/blob/main/data_sources/modules/dataforseo.py) processes one competitor domain at a time per method call. However, you can invoke the method multiple times with different competitor domains or wrap it in a loop to compare your rankings against several competitors sequentially, aggregating the results for a broader competitive landscape analysis.

### What is the difference between get_rankings() and get_serp_data()?

While both methods query the DataForSEO API, `get_rankings()` focuses on position tracking for a specific domain across multiple keywords, returning where that domain ranks and associated keyword metrics. In contrast, `get_serp_data()` retrieves a comprehensive snapshot of the entire SERP for a single keyword, including all organic results, SERP features like knowledge graphs or carousels, and detailed keyword intelligence such as competition level and CPC.