LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder: Data Structures and Algorithms Explained

The LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder utilizes a segmented circular buffer architecture with power-law bucketed edge pools and bit-masked edge types to store massive bipartite graphs with amortized O(1) insertions, O(k) neighbor retrievals, and strictly bounded memory footprints.

The LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder found in Twitter's open-source recommendation system (twitter/the-algorithm) functions as a thin Scala façade that constructs high-performance in-memory bipartite graphs. This builder delegates all heavy computational work to the GraphJet library while exposing a type-safe configuration interface for Twitter's real-time recommendation pipelines.

Core Data Structures

The builder itself is implemented in src/scala/com/twitter/recos/graph_common/LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder.scala, but the underlying storage mechanisms reside in GraphJet's core classes.

LeftIndexedPowerLawMultiSegmentBipartiteGraph

This class serves as the primary container for the entire bipartite graph structure. It maintains multiple time-based segments stored in a circular buffer, where each segment represents a slice of graph edges within a specific time window. When the active segment exceeds maxNumEdgesPerSegment, the allocator instantiates a new segment and evicts the oldest via FIFO replacement, ensuring constant-time insertion with bounded memory consumption.

PowerLawDegreeEdgePool

Each left node maintains its adjacency list in a PowerLawDegreeEdgePool, which implements a power-law bucketed array. Low-degree nodes occupy compact initial buckets while high-degree nodes receive exponentially larger bucket allocations based on the configured leftPowerLawExponent. This structure aligns memory allocation with the expected degree distribution observed in social graphs, minimizing overhead for the long tail of low-degree nodes while accommodating viral high-degree nodes efficiently.

EdgeTypeMask

Every edge stores a bit-mask (EdgeTypeMask) alongside the target node identifier, enabling fast categorical filtering during traversal. The mask supports operations like EdgeTypeMask.FOLLOW or EdgeTypeMask.FRIEND, allowing the graph to distinguish interaction types without requiring separate adjacency structures or additional memory indirection.

Segment Circular Buffer

The graph maintains segments in a circular buffer structure. Each Segment contains a collection of PowerLawDegreeEdgePool instances for the left side plus metadata including segment start timestamps and edge counters. This design yields O(1) segment allocation and eviction operations, critical for streaming ingestion pipelines that cannot tolerate GC pauses from large memory reallocations.

Algorithms and Complexity Guarantees

Segmented Memory Management with FIFO Eviction

When addEdge receives a new connection, the builder forwards it to the current active segment. If currentSegment.edgeCount >= maxNumEdgesPerSegment, the system allocates a fresh segment and drops the oldest segment from the circular buffer. This algorithm provides amortized O(1) insertion complexity while guaranteeing that memory consumption never exceeds maxNumSegments * maxNumEdgesPerSegment edges.

Power-Law Bucket Allocation Strategy

The PowerLawDegreeEdgePool computes bucket indices using the formula derived from leftPowerLawExponent and the node's current degree. Bucket sizes grow proportionally to degree^exponent, matching the natural power-law distribution of social network connectivity. Edge insertion requires locating the appropriate bucket (constant time) and writing the right-node ID plus edge-type mask into the bucket's backing array.

Edge Retrieval and Type Filtering

Neighbor queries scan all active segments and concatenate their respective PowerLawDegreeEdgePool instances. Because each pool stores edges in contiguous arrays, iteration executes as a tight loop with O(k) complexity where k represents the total degree across all segments. The stored EdgeTypeMask enables bit-wise filtering during this scan, eliminating the need for post-retrieval type checking or additional index structures.

Statistics Integration

The builder accepts a StatsReceiver parameter that propagates to the underlying graph implementation. GraphJet emits operational metrics including segment eviction counts and bucket overflow events, enabling real-time monitoring of graph construction performance and memory pressure.

Implementation Example: Building and Querying Graphs

The builder's apply method constructs the graph by forwarding configuration parameters directly to the GraphJet constructor:

def apply(
  graphBuilderConfig: GraphBuilderConfig,
  statsReceiverWrapper: StatsReceiver
): LeftIndexedPowerLawMultiSegmentBipartiteGraph = {
  new LeftIndexedPowerLawMultiSegmentBipartiteGraph(
    graphBuilderConfig.maxNumSegments,
    graphBuilderConfig.maxNumEdgesPerSegment,
    graphBuilderConfig.expectedNumLeftNodes,
    graphBuilderConfig.expectedMaxLeftDegree,
    graphBuilderConfig.leftPowerLawExponent,
    graphBuilderConfig.expectedNumRightNodes,
    graphBuilderConfig.edgeTypeMask,
    statsReceiverWrapper
  )
}

Building a Graph

import com.twitter.recos.graph_common._
import com.twitter.graphjet.bipartite.api.EdgeTypeMask
import com.twitter.stats.{StatsReceiver, StatsReceiverFactory}

val cfg = LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder.GraphBuilderConfig(
  maxNumSegments        = 5,
  maxNumEdgesPerSegment = 1_000_000,
  expectedNumLeftNodes  = 10_000_000,
  expectedMaxLeftDegree = 10_000,
  leftPowerLawExponent  = 2.5,
  expectedNumRightNodes = 50_000_000,
  edgeTypeMask          = EdgeTypeMask.FRIEND | EdgeTypeMask.FOLLOW
)

val stats: StatsReceiver = StatsReceiverFactory.defaultStatsReceiver
val graph = LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder(cfg, stats)

graph.addEdge(leftId = 12345L, rightId = 987654321L, edgeTypeMask = EdgeTypeMask.FOLLOW)

Querying Neighbors with Type Filtering

import scala.collection.mutable.ArrayBuffer

def neighbors(leftId: Long, mask: EdgeTypeMask): Seq[Long] = {
  val buffer = ArrayBuffer[Long]()
  graph.getSegments.foreach { segment =>
    val edges = segment.getEdges(leftId)
    edges.foreach { edge =>
      if ((edge.edgeTypeMask & mask) != 0) {
        buffer += edge.rightNodeId
      }
    }
  }
  buffer.toSeq
}

val followNeighbors = neighbors(12345L, EdgeTypeMask.FOLLOW)

Summary

• LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder acts as a configuration façade for GraphJet's native graph implementations.
• PowerLawDegreeEdgePool provides heterogeneous bucket sizing based on node degree, optimizing memory for power-law degree distributions.
• Circular buffer segmentation enables time-windowed graph storage with O(1) insertion and automatic FIFO eviction of stale data.
• EdgeTypeMask bit-masks allow categorical edge filtering during traversal without additional memory overhead.
• Amortized O(1) insertion complexity and O(k) retrieval complexity make the structure suitable for high-throughput recommendation systems.

Frequently Asked Questions

What is the time complexity of edge insertions in LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder?

Edge insertions operate in amortized O(1) time. The system appends edges to the current segment's PowerLawDegreeEdgePool until reaching maxNumEdgesPerSegment, at which point it allocates a new segment in constant time while evicting the oldest segment via the circular buffer mechanism.

How does the power-law exponent affect memory usage?

The leftPowerLawExponent parameter controls bucket growth in PowerLawDegreeEdgePool. Higher exponents allocate larger buckets to high-degree nodes more aggressively, reducing reallocation frequency for viral nodes while potentially increasing memory overhead. Lower exponents conserve memory but may trigger more frequent array resizing for high-degree nodes.

Where is the actual graph storage implemented?

While the builder resides in twitter/the-algorithm at src/scala/com/twitter/recos/graph_common/LeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder.scala, the actual storage implementation lives in the external GraphJet library. Specifically, LeftIndexedPowerLawMultiSegmentBipartiteGraph and PowerLawDegreeEdgePool are implemented in the twitter/graphjet repository.

What happens when all segments reach maximum capacity?

When the current active segment reaches maxNumEdgesPerSegment, the system automatically allocates a new segment and evicts the oldest segment from the circular buffer (FIFO eviction). This maintains a sliding time window of graph data and ensures memory usage remains bounded by maxNumSegments * maxNumEdgesPerSegment, regardless of total stream volume.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}