How SERP Data Fetching Works with Lazy-Loaded Sections in Open-SEO

Open-SEO fetches Google SERP data once per keyword and paginates results client-side using React state, displaying data in lazy-loaded 10-item sections without triggering additional network requests when users navigate between pages.

Open-SEO is an open-source SEO analytics platform that retrieves search engine results programmatically. The application implements an efficient SERP data fetching with lazy-loaded sections pattern that minimizes API costs while providing smooth pagination. This architecture ensures users can browse large result sets instantly after the initial data load.

Triggering the Initial SERP Data Fetch

The data retrieval process begins in the client-side hook useKeywordSerpAnalysis located at src/client/features/keywords/hooks/useKeywordSerpAnalysis.ts. When a user selects a keyword, the hook stores the selection in React state via setSerpKeyword, which enables a TanStack React-Query request.

The query function invokes the server-side endpoint getSerpAnalysis defined in src/serverFunctions/keywords.ts. This server function validates the input through serpAnalysisSchema and forwards the request to the keyword research service layer.

// Client hook triggering the fetch
const {
  setSerpKeyword,
  serpPage,
  setSerpPage,
  SERP_PAGE_SIZE,
  serpResults,
  serpLoading,
} = useKeywordSerpAnalysis(projectId, locationCode);

// Trigger the fetch by setting a keyword
setSerpKeyword('organic coffee');

Server-Side Processing and Caching

The server function getSerpAnalysis resolves the market configuration (location and language) before delegating to KeywordResearchService.getSerpAnalysis in src/server/features/keywords/services/KeywordResearchService.ts. The actual implementation resides in src/server/features/keywords/services/research/serp.ts.

This service layer implements a robust caching strategy:

1. Cache key generation – Builds a unique identifier based on keyword and market parameters.
2. R2 cache lookup – Checks Cloudflare R2 storage for existing results.
3. Live data fetch – If cache misses, calls createDataforseoClient(...).serp.live to retrieve fresh data from DataForSEO.
4. Result mapping – Filters to organic results using mapOrganicSerpItems.
5. Cache storage – Stores the SerpAnalysisResult for 12 hours to reduce API costs.

The returned SerpAnalysisResult type contains the original keyword and an array of SerpResultItem objects representing the organic search positions.

// Server function definition
export const getSerpAnalysis = createServerFn({ method: 'POST' })
  .middleware(requireProjectContext)
  .validator(serpAnalysisSchema)
  .handler(async ({ data, context }) =>
    KeywordResearchService.getSerpAnalysis(
      { ...data, ...resolveMarket(data, context.project), projectId: context.projectId },
      context,
    ),
  );

Client-Side Lazy Loading and Pagination

Once the server returns the full result array, the useKeywordSerpAnalysis hook stores all items in memory. The lazy-loaded sections behavior emerges through client-side pagination state management.

The hook maintains serpPage (current page index) and exposes SERP_PAGE_SIZE (set to 10 results per page). When users navigate between pages via setSerpPage, the UI updates instantly by slicing the in-memory array rather than requesting new data from the server. This architecture eliminates network latency during pagination and reduces external API usage costs.

The pagination logic ensures that all results remain available in React Query's cache, while only the current 10-item slice renders to the DOM.

Rendering the Paginated SERP Sections

The presentation layer receives the paginated data through KeywordResearchDesktopResults.tsx (and its mobile counterpart). These components pass the controller object from the hook into SerpAnalysisCard, which handles the visual slicing of results.

The card component receives the full items array along with page and pageSize props, then internally calculates the visible slice. This pattern decouples data fetching from presentation, allowing the UI to remain responsive while displaying the lazy-loaded sections.

// UI component implementing lazy-loaded pagination
<SerpAnalysisCard
  items={serpResults}
  page={serpPage}
  pageSize={SERP_PAGE_SIZE}
  onPageChange={setSerpPage}
  loading={serpLoading}
  error={serpError}
/>

Summary

• Single fetch architecture: Open-SEO retrieves complete SERP data once per keyword via getSerpAnalysis and caches it for 12 hours in R2 storage.
• Client-side pagination: The useKeywordSerpAnalysis hook manages serpPage state and SERP_PAGE_SIZE (10 items) to slice results without additional network requests.
• Service layer abstraction: KeywordResearchService.getSerpAnalysis coordinates between the DataForSEO API and internal caching in src/server/features/keywords/services/research/serp.ts.
• Lazy-loaded UI sections: Components like SerpAnalysisCard render only the current page slice while keeping the full dataset in memory for instant navigation.

Frequently Asked Questions

How does Open-SEO prevent redundant API calls when paginating SERP results?

Open-SEO fetches the complete result set once and stores it in TanStack React-Query's cache. The useKeywordSerpAnalysis hook manages pagination state through serpPage and slices the serpResults array client-side using SERP_PAGE_SIZE. When users change pages, setSerpPage updates the React state immediately without triggering the useQuery refetch, eliminating redundant calls to the DataForSEO API.

What is the cache duration for SERP data in Open-SEO?

The system caches SERP data for 12 hours in Cloudflare R2 storage. The src/server/features/keywords/services/research/serp.ts file implements this TTL (time-to-live) strategy to balance data freshness with API cost optimization, checking the cache before calling createDataforseoClient(...).serp.live.

Which components handle the lazy-loaded display of SERP sections?

The KeywordResearchDesktopResults.tsx file (and its mobile variant) orchestrates the layout by receiving the controller object from useKeywordSerpAnalysis. It passes pagination props to SerpAnalysisCard, which renders the current 10-item page slice while maintaining the full result set in memory for instantaneous page transitions.

How does the server function validate SERP analysis requests?

The getSerpAnalysis server function in src/serverFunctions/keywords.ts uses .validator(serpAnalysisSchema) to ensure incoming requests contain valid parameters. It also applies .middleware(requireProjectContext) to verify the user has appropriate access to the project before delegating to KeywordResearchService.getSerpAnalysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}