How protobuf-go-lite Uses fastjson for High-Performance JSON Parsing in Generated Code

The protobuf-go-lite json feature generates MarshalProtoJSON and UnmarshalProtoJSON methods that delegate tokenization to fastjson via the json-iterator-lite wrapper, delivering zero-allocation parsing for Go protobuf messages.

The aperturerobotics/protobuf-go-lite repository provides a lightweight, high-performance alternative to standard Go protobuf implementations. Its json feature leverages Valyala's fastjson library through the json-iterator-lite abstraction layer to generate optimized serialization code. This architecture minimizes heap allocations while maintaining full compatibility with the protobuf JSON specification.

Architecture of the fastjson Integration

The integration relies on a thin façade that exposes fastjson's optimized scanner to generated code without exposing low-level implementation details.

The json-iterator-lite Wrapper

According to the source code in json/plugin.go, the json feature imports json-iterator-lite (aliased as jsoniter), which internally wraps Valyala's fastjson. This wrapper provides a streaming API that the generated code uses for both reading and writing JSON tokens.

When unmarshaling begins, json/unmarshal.go creates an iterator via jsoniter.ParseBytes within the NewUnmarshalState function. This establishes the foundation for zero-copy parsing by utilizing fastjson's highly optimized byte scanner.

UnmarshalState and Token Parsing

The UnmarshalState struct defined in json/unmarshal.go encapsulates the fastjson-backed iterator. Primitive type readers such as ReadString, ReadInt32, and ReadFloat64 are thin wrappers around the underlying fastjson iterator methods. For example, when generated code calls s.ReadInt64(), the implementation forwards to fastjson's optimized number parsing routines.

Generated Unmarshaling Code

The code generator located in features/json/message-unmarshal.go emits UnmarshalProtoJSON methods that interact directly with the fastjson façade.

How UnmarshalProtoJSON Uses fastjson

Each generated message method receives an *jsonplugin.UnmarshalState parameter and uses it to traverse the JSON document. The generated code calls methods like s.ReadString() and s.ReadFloat64Array() to extract values. These calls translate directly to fastjson's tokenization engine, as shown in this representative generated code:

func (x *MyMessage) UnmarshalProtoJSON(s *jsonplugin.UnmarshalState) {
    // Fastjson-backed iterator reads the JSON object.
    s.ReadObject(func(key string) {
        switch key {
        case "id", "id":
            // Fast parsing of an int64, using fastjson under the hood.
            x.Id = s.ReadInt64()
        case "name", "name":
            // Fast parsing of a string.
            x.Name = s.ReadString()
        case "tags", "tags":
            // Fast parsing of a string array.
            x.Tags = s.ReadStringArray()
        default:
            // Unknown field – skip efficiently.
            s.Skip()
        }
    })
}

The heavy lifting of tokenization and value conversion remains inside fastjson, while the generated code handles protobuf-specific quirks such as wrapped primitives and field name aliasing.

Generated Marshaling Code

For serialization, features/json/message-marshal.go generates MarshalProtoJSON methods that utilize jsoniter.Stream for high-speed output.

Streaming JSON with MarshalState

The MarshalState (provided by json/stream.go) wraps a fastjson-backed stream writer. Generated code calls methods like s.WriteString, s.WriteInt64, and s.WriteArrayStart to emit JSON tokens. These calls translate directly to fastjson's high-speed output routines, enabling efficient streaming serialization without intermediate buffer allocations.

func (x *MyMessage) MarshalProtoJSON(s *jsonplugin.MarshalState) {
    s.WriteObjectStart()
    if x.Id != 0 {
        s.WriteObjectField("id")
        s.WriteInt64(x.Id) // fastjson stream writes the number.
    }
    if x.Name != "" {
        s.WriteObjectField("name")
        s.WriteString(x.Name) // fastjson stream writes the string.
    }
    if len(x.Tags) > 0 {
        s.WriteObjectField("tags")
        s.WriteArrayStart()
        for _, v := range x.Tags {
            s.WriteString(v) // fastjson stream writes each element.
        }
        s.WriteArrayEnd()
    }
    s.WriteObjectEnd()
}

Field Mask Support and Path Tracking

The UnmarshalState tracks the current JSON path using path and pathSlice fields to optionally build protobuf field masks. This logic operates independently of the parsing engine, simply wrapping the fastjson iterator. By preserving fastjson's performance characteristics while adding protobuf-specific metadata tracking, the implementation supports advanced features like selective field unmarshaling without sacrificing parsing speed.

Performance Benefits of fastjson Integration

The delegation of raw JSON handling to fastjson provides several advantages for generated code:

• Zero-allocation parsing: fastjson's scanner minimizes heap allocations during tokenization.
• Efficient skipping: The Skip() method uses fastjson's optimized skip logic to ignore unknown fields rapidly.
• Streaming writes: Marshaling uses fastjson's stream writer to emit JSON directly to the output buffer.
• Minimal abstraction overhead: The json-iterator-lite layer adds negligible latency compared to direct fastjson usage.

Summary

• The protobuf-go-lite json feature generates MarshalProtoJSON and UnmarshalProtoJSON methods for every protobuf message.
• Parsing delegates to fastjson through the json-iterator-lite wrapper, specifically via jsoniter.ParseBytes in json/unmarshal.go.
• Generated unmarshaling code in features/json/message-unmarshal.go calls readers like ReadInt64() and ReadString() that wrap fastjson's optimized token scanner.
• Generated marshaling code in features/json/message-marshal.go uses jsoniter.Stream via json/stream.go for high-speed JSON emission.
• Field mask support tracks JSON paths through UnmarshalState without interfering with fastjson's core parsing performance.

Frequently Asked Questions

What is the relationship between json-iterator-lite and fastjson in protobuf-go-lite?

The json-iterator-lite library is a lightweight façade that wraps Valyala's fastjson. In json/plugin.go, the code imports this library to provide the jsoniter.Iterator and jsoniter.Stream types used by the generated marshaling and unmarshaling code. This abstraction allows the generated methods to benefit from fastjson's zero-allocation parsing while maintaining a stable API surface.

How does UnmarshalState handle primitive type parsing?

The UnmarshalState struct defined in json/unmarshal.go creates a fastjson-backed iterator via NewUnmarshalState. Methods like ReadString(), ReadInt32(), and ReadFloat64() are thin wrappers that forward calls to the underlying fastjson iterator. This design ensures that primitive parsing utilizes fastjson's highly optimized number and string scanners while handling protobuf-specific requirements such as wrapped types.

Can the generated JSON marshaling code handle field masks?

Yes. The UnmarshalState tracks the current JSON path using internal path and pathSlice fields. As the fastjson iterator traverses the document, the state records field positions to optionally build a protobuf field mask. This capability operates alongside the fastjson parsing layer without degrading performance, allowing selective unmarshaling of specific fields.

Where does the actual fastjson parsing logic reside in the codebase?

The actual fastjson integration points are distributed across several files. json/unmarshal.go initializes the fastjson parser via jsoniter.ParseBytes, while features/json/message-unmarshal.go and features/json/message-marshal.go contain the generators that emit calls to this parser. The streaming writer implementation resides in json/stream.go, which uses the fastjson-backed jsoniter.Stream for output operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}