How dataforseo.py Retrieves SERP Positions and Keyword Metrics for Competitive Analysis

The dataforseo.py module acts as a thin Python client for the DataForSEO API, using environment-based Basic Auth to POST keyword tasks to /v3/serp/google/organic/live/advanced, then parsing the JSON response to extract exact SERP positions, search volume, CPC, and competitor gaps.

The dataforseo.py file, located in data_sources/modules/ within TheCraigHewitt/seomachine, provides the core integration with the DataForSEO API. This lightweight client enables the SEO Machine platform to retrieve live SERP positions and comprehensive keyword metrics necessary for data-driven competitive analysis.

Authentication and Session Management in dataforseo.py

Environment-Based Credentials

The DataForSEO class initializes by reading DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD from environment variables (lines 24-36). These credentials are encoded into a Basic Auth header using Base64, which the class attaches to a reusable requests.Session object stored as self.session.

The _post() Helper Method

All API calls flow through the private _post() method (lines 42-48). This helper accepts an endpoint path and a JSON payload, constructs the full URL by appending the path to the base URL (https://api.dataforseo.com), and returns the parsed JSON response. Using a single session ensures connection pooling and automatic header injection for every request.

Retrieving SERP Positions with get_rankings()

The get_rankings() method (lines 49-100) determines exactly where a target domain appears in Google’s organic results for a list of keywords.

Task Construction and API Endpoint

For each keyword provided, the method builds a task dictionary containing keyword, location_code, language_code, device, and os (lines 60-68). It then POSTs the entire task list to the /v3/serp/google/organic/live/advanced endpoint (line 78).

Parsing Positions and Metrics

Upon receiving the response, get_rankings() iterates through the tasks array. For each successful task (status code 20000), it extracts the items array holding organic results (lines 83-87). The method enumerates these items starting at 1, checking each domain field against the target domain. The first match records the position and URL (lines 88-95).

Finally, the method enriches each result with search volume and CPC data pulled from keyword_data.keyword_info (lines 99-104), returning a list of dictionaries containing keyword, domain, position, url, ranking (boolean), search_volume, and cpc.

from data_sources.modules.dataforseo import DataForSEO

dfs = DataForSEO()                     # reads credentials from .env

rankings = dfs.get_rankings(
    domain="example.com",
    keywords=["seo tools", "keyword research", "content optimization"]
)

for r in rankings:
    print(f"{r['keyword']}: position={r['position'] or '–'}, volume={r['search_volume']}")

Extracting Full Keyword Metrics via get_serp_data()

While get_rankings() focuses on position tracking, get_serp_data() (lines 108-174) retrieves a comprehensive snapshot of a single keyword’s SERP landscape.

Organic Results and SERP Features

The method constructs a single-keyword task with a configurable depth parameter (lines 125-132) and POSTs to the same organic endpoint. After validating the response (lines 136-142), it parses:

• organic_results: A list of dictionaries containing position, url, domain, title, description, and breadcrumb (lines 146-157).
• features: Non-organic SERP elements such as “carousel” or “knowledge_graph” collected from the result items (lines 158-163).

Keyword-Level Intelligence

Beyond the SERP structure, the method extracts quantitative metrics from keyword_data.keyword_info: search_volume, cpc, and competition (lines 164-170). These values enable the SEO Machine to calculate opportunity scores and prioritize keywords based on traffic potential and cost-per-click data.

serp = dfs.get_serp_data("seo automation", limit=50)

print("Search volume:", serp["search_volume"])
print("CPC:", serp["cpc"])
print("Top organic result:", serp["organic_results"][0]["url"])
print("SERP features:", ", ".join(serp["features"]))

Competitive Gap Analysis with analyze_competitor()

The analyze_competitor() method (lines 176-233) leverages the previous methods to perform side-by-side ranking comparisons.

Position Gap Calculation

For each keyword in the analysis set, the method retrieves SERP data and records the position of both the competitor domain and your domain (lines 213-219). It calculates the gap as your_position - competitor_position, where negative values indicate the competitor outranks you.

Opportunity Scoring

Based on the gap size and whether you rank at all, the method assigns an opportunity level (high, medium, low) to each keyword (line 231). This scoring system allows SEO teams to prioritize keywords where competitors have significant advantages but search volume justifies the effort to close the gap.

comparison = dfs.analyze_competitor(
    competitor_domain="competitor.com",
    keywords=["seo audit", "link building", "technical seo"],
    your_domain="example.com"
)

for entry in comparison["comparison"]:
    print(f"{entry['keyword']}: competitor={entry['competitor_position']}, "
          f"you={entry['your_position']}, gap={entry['gap']}, "
          f"opportunity={entry['opportunity']}")

Summary

• dataforseo.py acts as a thin Python client for the DataForSEO API, handling authentication via environment variables and a reusable requests.Session.
• The get_rankings() method retrieves exact SERP positions for a target domain across multiple keywords, enriching results with search volume and CPC data.
• get_serp_data() provides comprehensive keyword intelligence, including organic results, SERP features, and competitive metrics like competition level.
• analyze_competitor() compares your rankings against competitors, calculating position gaps and assigning opportunity scores to prioritize SEO efforts.

Frequently Asked Questions

How does dataforseo.py handle API authentication?

The DataForSEO class reads DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD from environment variables during initialization. It encodes these credentials into a Basic Auth header using Base64 and attaches them to a reusable requests.Session object, ensuring every subsequent API request includes proper authentication automatically.

What SERP data does the get_rankings() method return?

The get_rankings() method returns a list of dictionaries, each containing the target keyword, domain, position (the exact SERP index where the domain appears), the ranking url, a boolean ranking flag, plus keyword metrics including search_volume and cpc pulled from the DataForSEO API's keyword_info object.

Can dataforseo.py analyze multiple competitors simultaneously?

The current implementation of analyze_competitor() in data_sources/modules/dataforseo.py processes one competitor domain at a time per method call. However, you can invoke the method multiple times with different competitor domains or wrap it in a loop to compare your rankings against several competitors sequentially, aggregating the results for a broader competitive landscape analysis.

What is the difference between get_rankings() and get_serp_data()?

While both methods query the DataForSEO API, get_rankings() focuses on position tracking for a specific domain across multiple keywords, returning where that domain ranks and associated keyword metrics. In contrast, get_serp_data() retrieves a comprehensive snapshot of the entire SERP for a single keyword, including all organic results, SERP features like knowledge graphs or carousels, and detailed keyword intelligence such as competition level and CPC.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}