Openship Real-Time Log Streaming for Build Deployments: SSE Architecture and Base64 Encoding

Openship captures every build step output—from npm install to Docker build—and streams it live to browsers using Server-Sent Events (SSE) with base64-encoded payloads, while simultaneously persisting logs to the database for historical replay.

Openship provides instant deployment visibility through a multi-layered log streaming pipeline that bridges cloud executors, bare-metal servers, and Docker environments with user interfaces. This article examines the complete technical architecture of Openship's real-time log streaming system, from adapter-level capture to browser rendering, as implemented in the oblien/openship repository.

Three-Layer Real-Time Log Streaming Architecture

The Openship build pipeline processes deployment output through three distinct layers that ensure every log entry reaches the user's terminal with minimal latency.

Adapter Execution and Log Capture

The build driver—whether using Cloud Oblien, Bare Executor, or Docker—generates log entries during command execution. Cloud adapters emit base64-encoded chunks natively over SSE connections, while Bare and Docker adapters produce plain-text lines that the system encodes later in the pipeline. All output is wrapped in a standardized LogEntry interface containing a message field and an optional rawData property for pre-encoded binary streams.

According to the implementation details in packages/adapters/docs/LOG-STREAMING.md, this abstraction allows disparate execution environments to feed into a unified streaming interface without protocol fragmentation.

Session Management and Base64 Encoding

The logCallback function, implemented in apps/api/src/modules/deployments/build.service.ts, serves as the central ingestion point for all build output. When a LogEntry arrives, the callback performs two critical operations: it persists the entry to a database array for long-term storage, and it appends the log to an in-memory session via sessionManager.appendLog(sessionId, entry).

If the entry contains rawData from a Cloud adapter, the system passes it through unchanged. Otherwise, the plain-text message is base64-encoded at the hand-off point before broadcast. The resulting SSE payload follows this structure:

{
  "type": "log",
  "data": "<base64-encoded-chunk>",
  "eventId": "uuid",
  "step": "build",
  "level": "info"
}

The session.manager.ts module maintains these in-memory log sessions and manages the SSE subscriber list, ensuring that multiple connected clients receive identical streams simultaneously.

Browser Consumption and Terminal Rendering

The frontend initiates an EventSource connection to receive the SSE stream, typically through the useSSEStream.ts hook located in apps/web/src/hooks/. Each incoming chunk undergoes a specific decoding sequence: atob converts the base64 string to a binary string, Uint8Array.from transforms it into a byte array, and TextDecoder produces the final UTF-8 text. The decoded output is then written directly to an xterm.js terminal instance for display.

For historical builds that have already completed, the system retrieves stored logs via GET /api/deployments/:id/build-status and replays them through the same terminal component using useDeploymentBuild.ts.

Why Base64 Encoding is Required for SSE Log Streaming

Build output frequently contains ANSI escape codes, carriage returns, and other binary-unfriendly characters that can corrupt text-only transport channels. By encoding the stream as base64, Openship guarantees safe transmission over SSE—a text-based protocol—without accidental escaping or truncation. This approach ensures that color codes, progress bars, and binary artifacts in the build logs render identically in the browser to how they appear in local terminals.

Server-Side Implementation: Capturing and Broadcasting Logs

The core server-side logic resides in apps/api/src/modules/deployments/build.service.ts, where the logCallback function handles both persistence and real-time distribution:

// inside apps/api/src/modules/deployments/build.service.ts
function logCallback(entry: LogEntry) {
  // Persist for later replay
  logs.push(entry);                       // DB array
  // Keep it in‑memory for live sessions
  sessionManager.appendLog(sessionId, entry);
}

This dual-write strategy ensures that users connecting mid-deployment receive historical context immediately, while live subscribers see new entries within milliseconds of generation.

Client-Side Implementation: Consuming the SSE Stream

The frontend hook useSSEStream.ts manages the EventSource lifecycle and decoding pipeline:

// inside apps/web/src/hooks/useSSEStream.ts
const source = new EventSource(`/api/deployments/${id}/logs`);
source.onmessage = ev => {
  const bytes = Uint8Array.from(atob(ev.data), c => c.charCodeAt(0));
  const text = new TextDecoder().decode(bytes);
  terminal.write(text);
};

For completed deployments, the historical replay mechanism in useDeploymentBuild.ts fetches the entire log history and feeds it line-by-line into the same terminal instance:

// apps/web/src/hooks/useDeploymentBuild.ts
fetch(`/api/deployments/${id}/build-status`)
  .then(r => r.json())
  .then(data => {
    data.logs.split('\n')
      .filter(l => l.trim())
      .forEach(line => terminal.write(line + '\r\n'));
  });

Key Source Files and Their Roles

• packages/adapters/docs/LOG-STREAMING.md – Comprehensive documentation of the streaming pipeline, encoding requirements, and database persistence strategy.
• apps/api/src/modules/deployments/build.service.ts – Core build service that implements logCallback, creates LogEntry objects, and hands them to the session manager.
• apps/api/src/modules/deployments/session.manager.ts – Manages in-memory log sessions, appends new entries, and broadcasts to SSE subscribers.
• apps/web/src/hooks/useSSEStream.ts – Frontend hook that opens the SSE connection, decodes base64 payloads, and writes to the xterm.js terminal.
• apps/web/src/hooks/useDeploymentBuild.ts – Retrieves historical logs for completed builds and replays them in the terminal component.

Summary

• Openship uses Server-Sent Events (SSE) to deliver real-time build logs from execution environments to browser terminals with minimal overhead.
• Base64 encoding ensures binary-safe transmission of ANSI escape codes and special characters over the text-based SSE protocol.
• The logCallback function in build.service.ts provides dual-write durability, persisting logs to the database while broadcasting to in-memory sessions.
• Cloud adapters emit pre-encoded base64 chunks, while Bare and Docker adapters rely on the platform to encode plain text at the session management layer.
• Historical replay fetches stored logs via the build-status API endpoint and feeds them through the same xterm.js terminal component used for live streaming.

Frequently Asked Questions

How does Openship handle log streaming for different execution environments?

Openship abstracts environment differences through adapter-specific implementations. Cloud adapters natively emit base64-encoded chunks over SSE, while Bare Executor and Docker adapters produce plain text that build.service.ts encodes before broadcasting. All adapters conform to the LogEntry interface, ensuring the session manager receives a uniform data structure regardless of the underlying infrastructure.

Why does Openship use base64 encoding instead of plain text for SSE?

Base64 encoding prevents corruption of build output during transmission over SSE, which is strictly a text-based protocol. Build logs frequently contain ANSI escape sequences for colors, carriage returns for progress bars, and other non-printable characters. Encoding these as base64 guarantees that the browser receives the exact byte sequence intended by the build process, avoiding truncation or malformation that could occur with raw text transmission.

How can developers access historical build logs after deployment completes?

Completed build logs remain accessible through the GET /api/deployments/:id/build-status endpoint, which retrieves the persisted log entries from the database. The frontend hook useDeploymentBuild.ts fetches this data and replays it through the xterm.js terminal component, allowing developers to review past deployments with the same formatting and color coding they saw during the live stream.

What frontend technologies power the terminal interface in Openship?

The frontend utilizes the native browser EventSource API to maintain persistent SSE connections, combined with the xterm.js library for terminal emulation. Incoming base64 payloads undergo decoding via atob and TextDecoder before being written to the terminal. This architecture supports both live streaming and historical replay within the same interactive terminal component.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}