How Nitter's parseGraphTimeline Function Handles GraphQL Timeline Entries

Nitter's parseGraphTimeline function in src/parser.nim converts raw Twitter GraphQL JSON into a structured Profile object by iterating through instruction arrays, extracting tweets from module items and entries, parsing conversation threads, and capturing pagination cursors and pinned tweets.

The parseGraphTimeline function serves as the core parsing engine in the zedeus/nitter repository, transforming chaotic Twitter GraphQL responses into clean, typed Nim data structures. This procedural powerhouse located in src/parser.nim handles the complex logic required to identify timeline entries, separate individual tweets from conversation threads, and manage pagination state across user timelines.

Core Implementation in src/parser.nim

Located at lines 78-114 of src/parser.nim, the parseGraphTimeline function signature accepts a JsonNode and an optional after cursor string, returning a fully populated Profile object. The function initializes an empty Profile with a Timeline structure, setting the beginning flag to true only when the after parameter is empty, indicating the first page of results.

Step-by-Step Entry Processing

The parser employs a multi-stage extraction strategy to handle Twitter's nested instruction format.

Locating the Instruction List

Twitter's GraphQL payload nests timeline data within a list of "instructions" that drive the rendering logic. The parser attempts three distinct JSON paths to locate these instructions—searching for list-timeline, user-timeline, or user-result structures. If no valid instruction array is found, the function immediately returns the empty Profile, preventing downstream errors.

Handling Module Items

When an instruction contains a moduleItems array (typically marked as "AddToModule" types), the parser delegates to extractTweetsFromModuleItems. This helper extracts every tweet from these modular entries and appends them directly to result.tweets.content. After processing module items, the loop continues to the next instruction, skipping standard entry processing for that iteration.

Processing Standard Entries

For instructions containing an entries array, the parser examines each entryId to determine the content type:

• Tweet entries: Identified by entryId prefixes of tweet or profile-grid, these are processed via extractTweetsFromEntry and added to the timeline content.
• Conversation entries: Marked by -conversation- or homeConversation in the entryId, these trigger parseGraphThread to reconstruct full conversation threads before adding them to the timeline.
• Pagination cursors: Entries with entryId starting with cursor-bottom capture the next page token, stored in result.tweets.bottom for subsequent requests.

Detecting Pinned Tweets

On the initial page load (when after.len == 0), the parser checks for TimelinePinEntry instructions. When found, it extracts the first tweet, marks it with pinned = true, and stores it in result.pinned as an optional value, mirroring Twitter's pinned tweet behavior exactly.

Data Structures and Type Definitions

The parser relies on type definitions from src/types.nim, specifically the Profile object containing a tweets field of type Timeline. The Timeline structure maintains content (a sequence of Tweet objects), bottom (pagination cursor), and beginning (boolean flag). These types enable type-safe processing throughout Nitter's routing and view layers.

Integration with the Request Lifecycle

The parseGraphTimeline function operates within a clear separation of concerns:

1. Fetching: HTTP handlers in src/routes/timeline.nim call fetchProfile, which uses getGraphUserTweets to retrieve raw JSON from Twitter's GraphQL endpoints.
2. Parsing: The JSON passes to parseGraphTimeline, which produces a Profile containing structured timeline data.
3. Rendering: src/views/timeline.nim consumes the Profile to generate the final HTML output presented to users.

Practical Usage Examples

Basic parsing of GraphQL response data:

import parser, packedjson

# rawJson obtained from Twitter's GraphQL endpoint

let profile = parseGraphTimeline(rawJson)

echo "Fetched tweets: ", profile.tweets.content.len
if profile.pinned.isSome:
  echo "Pinned tweet ID: ", profile.pinned.get.id
echo "Next cursor: ", profile.tweets.bottom

Async integration within route handlers:

proc getUserTimeline(userId: string, after = ""): Future[Profile] {.async.} =
  let json = await getGraphUserTweets(userId, TimelineKind.tweets, after)
  return parseGraphTimeline(json, after)

Summary

• Location: The parseGraphTimeline function resides in src/parser.nim and serves as the primary GraphQL-to-Nim transformer.
• Input: Accepts JsonNode from Twitter's GraphQL API and an optional pagination cursor.
• Processing: Iterates instruction arrays, handling moduleItems and entries separately to extract tweets and threads.
• Output: Returns a Profile object containing a Timeline with content, pagination cursors, and optional pinned tweets.
• Architecture: Maintains clean separation between network fetching, JSON parsing, and HTML rendering layers.

Frequently Asked Questions

What data structure does parseGraphTimeline return?

The function returns a Profile object defined in src/types.nim, containing a tweets field of type Timeline. This timeline includes a sequence of Tweet objects, a bottom string for pagination cursors, a beginning boolean flag, and an optional pinned field for the user's pinned tweet.

How does parseGraphTimeline handle different types of timeline entries?

The parser distinguishes entry types by examining the entryId string prefix. Entries starting with tweet or profile-grid are processed as individual tweets, those containing -conversation- or homeConversation trigger thread parsing via parseGraphThread, and cursor-bottom entries capture pagination tokens for subsequent requests.

What happens if the GraphQL response lacks instruction data?

If the parser cannot locate valid instructions through any of the three supported JSON paths (list-timeline, user-timeline, or user-result), the function immediately returns an empty Profile object. This defensive programming prevents null pointer exceptions and allows upstream handlers to detect missing data gracefully.

Where does parseGraphTimeline fit in Nitter's request handling flow?

The function sits between the network layer and the view layer. HTTP routes in src/routes/timeline.nim fetch raw JSON using getGraphUserTweets, pass it to parseGraphTimeline for transformation into typed objects, and then hand the resulting Profile to src/views/timeline.nim for HTML rendering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}