# How SERP Data Fetching Works with Lazy-Loaded Sections in Open-SEO

> Learn how Open-SEO fetches Google SERP data efficiently. Discover client-side pagination and lazy-loaded sections for faster insights without extra network requests.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-07

---

**Open-SEO fetches Google SERP data once per keyword and paginates results client-side using React state, displaying data in lazy-loaded 10-item sections without triggering additional network requests when users navigate between pages.**

Open-SEO is an open-source SEO analytics platform that retrieves search engine results programmatically. The application implements an efficient **SERP data fetching with lazy-loaded sections** pattern that minimizes API costs while providing smooth pagination. This architecture ensures users can browse large result sets instantly after the initial data load.

## Triggering the Initial SERP Data Fetch

The data retrieval process begins in the client-side hook `useKeywordSerpAnalysis` located at [`src/client/features/keywords/hooks/useKeywordSerpAnalysis.ts`](https://github.com/every-app/open-seo/blob/main/src/client/features/keywords/hooks/useKeywordSerpAnalysis.ts). When a user selects a keyword, the hook stores the selection in React state via `setSerpKeyword`, which enables a TanStack React-Query request.

The query function invokes the server-side endpoint `getSerpAnalysis` defined in [`src/serverFunctions/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/keywords.ts). This server function validates the input through `serpAnalysisSchema` and forwards the request to the keyword research service layer.

```typescript
// Client hook triggering the fetch
const {
  setSerpKeyword,
  serpPage,
  setSerpPage,
  SERP_PAGE_SIZE,
  serpResults,
  serpLoading,
} = useKeywordSerpAnalysis(projectId, locationCode);

// Trigger the fetch by setting a keyword
setSerpKeyword('organic coffee');

```

## Server-Side Processing and Caching

The server function `getSerpAnalysis` resolves the market configuration (location and language) before delegating to `KeywordResearchService.getSerpAnalysis` in [`src/server/features/keywords/services/KeywordResearchService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/keywords/services/KeywordResearchService.ts). The actual implementation resides in [`src/server/features/keywords/services/research/serp.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/keywords/services/research/serp.ts).

This service layer implements a robust caching strategy:

1. **Cache key generation** – Builds a unique identifier based on keyword and market parameters.
2. **R2 cache lookup** – Checks Cloudflare R2 storage for existing results.
3. **Live data fetch** – If cache misses, calls `createDataforseoClient(...).serp.live` to retrieve fresh data from DataForSEO.
4. **Result mapping** – Filters to organic results using `mapOrganicSerpItems`.
5. **Cache storage** – Stores the `SerpAnalysisResult` for 12 hours to reduce API costs.

The returned `SerpAnalysisResult` type contains the original keyword and an array of `SerpResultItem` objects representing the organic search positions.

```typescript
// Server function definition
export const getSerpAnalysis = createServerFn({ method: 'POST' })
  .middleware(requireProjectContext)
  .validator(serpAnalysisSchema)
  .handler(async ({ data, context }) =>
    KeywordResearchService.getSerpAnalysis(
      { ...data, ...resolveMarket(data, context.project), projectId: context.projectId },
      context,
    ),
  );

```

## Client-Side Lazy Loading and Pagination

Once the server returns the full result array, the `useKeywordSerpAnalysis` hook stores all items in memory. The **lazy-loaded sections** behavior emerges through client-side pagination state management.

The hook maintains `serpPage` (current page index) and exposes `SERP_PAGE_SIZE` (set to 10 results per page). When users navigate between pages via `setSerpPage`, the UI updates instantly by slicing the in-memory array rather than requesting new data from the server. This architecture eliminates network latency during pagination and reduces external API usage costs.

The pagination logic ensures that all results remain available in React Query's cache, while only the current 10-item slice renders to the DOM.

## Rendering the Paginated SERP Sections

The presentation layer receives the paginated data through [`KeywordResearchDesktopResults.tsx`](https://github.com/every-app/open-seo/blob/main/KeywordResearchDesktopResults.tsx) (and its mobile counterpart). These components pass the controller object from the hook into `SerpAnalysisCard`, which handles the visual slicing of results.

The card component receives the full `items` array along with `page` and `pageSize` props, then internally calculates the visible slice. This pattern decouples data fetching from presentation, allowing the UI to remain responsive while displaying the lazy-loaded sections.

```tsx
// UI component implementing lazy-loaded pagination
<SerpAnalysisCard
  items={serpResults}
  page={serpPage}
  pageSize={SERP_PAGE_SIZE}
  onPageChange={setSerpPage}
  loading={serpLoading}
  error={serpError}
/>

```

## Summary

- **Single fetch architecture**: Open-SEO retrieves complete SERP data once per keyword via `getSerpAnalysis` and caches it for 12 hours in R2 storage.
- **Client-side pagination**: The `useKeywordSerpAnalysis` hook manages `serpPage` state and `SERP_PAGE_SIZE` (10 items) to slice results without additional network requests.
- **Service layer abstraction**: `KeywordResearchService.getSerpAnalysis` coordinates between the DataForSEO API and internal caching in [`src/server/features/keywords/services/research/serp.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/keywords/services/research/serp.ts).
- **Lazy-loaded UI sections**: Components like `SerpAnalysisCard` render only the current page slice while keeping the full dataset in memory for instant navigation.

## Frequently Asked Questions

### How does Open-SEO prevent redundant API calls when paginating SERP results?

Open-SEO fetches the complete result set once and stores it in TanStack React-Query's cache. The `useKeywordSerpAnalysis` hook manages pagination state through `serpPage` and slices the `serpResults` array client-side using `SERP_PAGE_SIZE`. When users change pages, `setSerpPage` updates the React state immediately without triggering the `useQuery` refetch, eliminating redundant calls to the DataForSEO API.

### What is the cache duration for SERP data in Open-SEO?

The system caches SERP data for **12 hours** in Cloudflare R2 storage. The [`src/server/features/keywords/services/research/serp.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/keywords/services/research/serp.ts) file implements this TTL (time-to-live) strategy to balance data freshness with API cost optimization, checking the cache before calling `createDataforseoClient(...).serp.live`.

### Which components handle the lazy-loaded display of SERP sections?

The [`KeywordResearchDesktopResults.tsx`](https://github.com/every-app/open-seo/blob/main/KeywordResearchDesktopResults.tsx) file (and its mobile variant) orchestrates the layout by receiving the controller object from `useKeywordSerpAnalysis`. It passes pagination props to `SerpAnalysisCard`, which renders the current 10-item page slice while maintaining the full result set in memory for instantaneous page transitions.

### How does the server function validate SERP analysis requests?

The `getSerpAnalysis` server function in [`src/serverFunctions/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/keywords.ts) uses `.validator(serpAnalysisSchema)` to ensure incoming requests contain valid parameters. It also applies `.middleware(requireProjectContext)` to verify the user has appropriate access to the project before delegating to `KeywordResearchService.getSerpAnalysis`.