Core Utilities in Graph Common for Building GraphJet-Based Graphs with Power-Law Degree Distributions

The Graph Common package provides specialized builders and wrappers—such as MultiSegmentPowerLawBipartiteGraphBuilder and FinagleStatsReceiverWrapper—that enable the construction of in-memory, power-law distributed bipartite graphs with minimal boilerplate and full observability integration.

The graph_common package located at src/scala/com/twitter/recos/graph_common within the Twitter recommendation algorithm repository supplies the foundational components for creating robust GraphJet-based graphs. These core utilities encapsulate graph-initialization parameters and GraphJet-specific details, allowing Recos services to efficiently model user-item relationships—such as user-video, user-tweet, and user-user interactions—as multi-segment structures exhibiting power-law degree distributions.

Multi-Segment Power-Law Graph Builders

MultiSegmentPowerLawBipartiteGraphBuilder

Located in MultiSegmentPowerLawBipartiteGraphBuilder.scala, this utility creates a MultiSegmentPowerLawBipartiteGraph that automatically shards the graph into time-based segments while preserving power-law characteristics. The builder uses a GraphBuilderConfig case class to bundle initialization parameters including maxNumSegments, expectedNumLeftNodes, leftPowerLawExponent, and degree caps. The apply(config, statsReceiver) method instantiates the graph with an ActionEdgeTypeMask and a wrapped stats receiver, handling internal segment management transparently.

NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder

For scenarios requiring rich node features, NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder.scala provides a builder that attaches metadata to right nodes. This is essential when downstream recommendation models require embeddings or feature vectors per item. The configuration extends the base builder with numRightNodeMetadataTypes and a custom EdgeTypeMask, returning a NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraph optimized for left-indexed queries with right-side attributes.

RightNodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder

The builder defined in RightNodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder.scala mirrors the metadata functionality but structures the graph to store metadata on the right side while maintaining left indexing. This variant supports Recos pipelines where right-side entities carry per-item features that must remain accessible during graph traversal operations.

Observability and Metrics Integration

FinagleStatsReceiverWrapper and FinagleCounterWrapper

GraphJet requires a StatsReceiver interface for monitoring, but Twitter's infrastructure relies on Finagle. The FinagleStatsReceiverWrapper class in FinagleStatsReceiverWrapper.scala adapts Finagle's StatsReceiver to GraphJet's interface, implementing scope and counter methods that delegate to Finagle's telemetry pipeline. Complementing this, FinagleCounterWrapper in FinagleCounterWrapper.scala wraps Finagle counters to satisfy GraphJet's Counter interface, ensuring that metrics like edge insertions and segment drops emit to standard Twitter monitoring systems without direct Finagle dependencies in the graph logic.

Edge Classification and Graph Helpers

ActionEdgeTypeMask

The ActionEdgeTypeMask class in ActionEdgeTypeMask.scala implements GraphJet's EdgeTypeMask interface to classify edge actions such as follows, retweets, or likes. This mask provides the mask logic used by all graph builders to distinguish between interaction types within the same bipartite structure, enabling weighted recommendation signals without requiring separate graph instances.

BipartiteGraphHelper and NodeInfoHandler

BipartiteGraphHelper.scala contains static utility functions for common graph operations including getDegree queries and power-law validation checks. For persistence operations, NodeInfoHandler.scala supplies writeNodeInfo and readNodeInfo methods that handle serialization and deserialization of node metadata, ensuring that graph state can be saved and restored without losing power-law configuration or node attributes.

Configuration and Implementation Examples

Constructing a power-law graph requires defining the degree distribution parameters through the builder configuration case classes. The following examples demonstrate standard patterns used in Twitter's Recos services.

Basic multi-segment power-law graph:

import com.twitter.recos.graph_common._
import com.twitter.graphjet.stats.StatsReceiver
import com.twitter.finagle.stats.{StatsReceiver => FinagleStats}

val cfg = MultiSegmentPowerLawBipartiteGraphBuilder.GraphBuilderConfig(
  maxNumSegments = 10,
  maxNumEdgesPerSegment = 10000000,
  expectedNumLeftNodes = 5000000,
  expectedMaxLeftDegree = 500,
  leftPowerLawExponent = 2.5,
  expectedNumRightNodes = 10000000,
  expectedMaxRightDegree = 200,
  rightPowerLawExponent = 2.0
)

val finagleStats: FinagleStats = ???
val graph = MultiSegmentPowerLawBipartiteGraphBuilder(
  cfg,
  FinagleStatsReceiverWrapper(finagleStats)
)

Graph with right-node metadata:

val metaCfg = NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder.GraphBuilderConfig(
  maxNumSegments = 8,
  maxNumEdgesPerSegment = 5000000,
  expectedNumLeftNodes = 3000000,
  expectedMaxLeftDegree = 400,
  leftPowerLawExponent = 2.2,
  expectedNumRightNodes = 8000000,
  numRightNodeMetadataTypes = 2,  // e.g., embedding + category
  edgeTypeMask = new ActionEdgeTypeMask()
)

val metaGraph = NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder(
  metaCfg,
  FinagleStatsReceiverWrapper(finagleStats)
)

Accessing metrics via the wrapper:

val segmentCounter = FinagleStatsReceiverWrapper(finagleStats).counter("segments_created")
segmentCounter.incr()

Summary

• Multi-segment builders like MultiSegmentPowerLawBipartiteGraphBuilder automate the creation of time-sharded, power-law distributed bipartite graphs through configuration-driven initialization in src/scala/com/twitter/recos/graph_common/.
• Metadata-aware variants extend base graphs with right-node or left-node feature storage, enabling rich recommendation contexts without altering core graph topology.
• Finagle integration wrappers bridge GraphJet's monitoring interfaces with Twitter's telemetry infrastructure, exposing counters and stats for operational visibility.
• Edge classification and helper utilities provide action-type masking via ActionEdgeTypeMask and operational methods for degree queries and serialization via BipartiteGraphHelper and NodeInfoHandler.

Frequently Asked Questions

What is the purpose of the GraphBuilderConfig case class in Graph Common?

The GraphBuilderConfig case class encapsulates all initialization parameters required to construct a power-law bipartite graph, including the number of segments, expected node counts, maximum degrees, and power-law exponents for both left and right nodes. This configuration object ensures type-safe, immutable parameter passing to the builder's apply method.

How does Graph Common handle monitoring and statistics for GraphJet graphs?

Graph Common adapts Twitter's Finagle metrics to GraphJet's interface through FinagleStatsReceiverWrapper and FinagleCounterWrapper. These classes delegate GraphJet's statistical operations—such as segment creation counters and edge insertion rates—to Finagle's StatsReceiver, allowing seamless integration with Twitter's existing monitoring dashboards.

When should I use NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder instead of the standard builder?

Use NodeMetadataLeftIndexedPowerLawMultiSegmentBipartiteGraphBuilder when your recommendation pipeline requires storing auxiliary data—such as embeddings, categories, or feature vectors—directly on right nodes. This builder extends the standard power-law graph with numRightNodeMetadataTypes support, eliminating the need for external lookup tables during graph traversal.

What role does ActionEdgeTypeMask play in graph construction?

ActionEdgeTypeMask implements GraphJet's EdgeTypeMask interface to distinguish between different interaction types within the same graph structure. It assigns bit-masked identifiers to actions like follows or retweets, enabling the graph to store heterogeneous relationships while maintaining a single, memory-efficient bipartite representation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}