How to Read Files Efficiently with mulle-buffer: 5 High-Performance Techniques

Use mulle_buffer_guarantee to reserve writable space for file chunks, read directly into the buffer with POSIX read(2), and commit bytes with mulle_buffer_advance to eliminate double-copying, or wrap memory-mapped regions using MULLE_BUFFER_STATIC_DATA for true zero-copy file access.

The mulle-c/mulle-buffer library provides a lightweight, growable-memory abstraction designed as a building block for high-performance I/O. While it does not expose a dedicated "read file" API, the headers src/mulle-buffer.h and src/mulle-flushablebuffer.h contain all primitives needed to implement efficient file readers with minimal overhead.

Core Primitives for Zero-Copy and Streaming Reads

Dynamic Growth with Guaranteed Capacity

When file sizes are unknown, dynamic growth prevents over-allocation while ensuring contiguous storage. In src/mulle-buffer.h, the mulle_buffer_guarantee function reserves capacity without advancing the write pointer, allowing direct I/O into uninitialized memory.

Call mulle_buffer_guarantee(buf, n) to obtain a writable pointer for the next n bytes, fill it with read(2) or fread, then commit the actual byte count with mulle_buffer_advance. This approach eliminates temporary buffers and reduces system call overhead by reading in large chunks (e.g., 8 KB).

Static Storage for Memory-Mapped Files

For scenarios requiring true zero-copy access, in-place static storage wraps existing memory blocks without allocation. The macro MULLE_BUFFER_STATIC_DATA (or the function mulle_buffer_init_with_static_bytes) in src/mulle-buffer.h initializes a buffer from an external memory region.

Use this with mmap(2) on POSIX or CreateFileMapping / MapViewOfFile on Windows. The buffer references the mapped region directly—no memcpy occurs during initialization.

Read-Only Safety and Zero-Copy Extraction

After loading completes, read-only mode prevents accidental modifications and enables safe concurrent access. The function mulle_buffer_set_readonly (declared in src/mulle-buffer.h) locks the buffer state.

For parsing without memory duplication, mulle_buffer_reference_bytes returns a direct pointer to internal storage. Feed this pointer to tokenizers, UTF-8 validators, or other byte-oriented parsers without extracting or copying data.

Streaming with Flushable Buffers

The flushable buffer variant in src/mulle-flushablebuffer.h supports custom read callbacks through the mulle_flushablebuffer_flusher_t hook. Implement a flusher that reads from FILE* handles or network sockets, appending data to the underlying buffer without intermediate temporaries.

This pattern enables streaming reads where the buffer grows on-demand as new chunks arrive, ideal for processing files larger than physical memory.

Byte-Level Navigation

For parsers requiring sequential scanning, src/mulle-buffer.h exposes bounded read operations: mulle_buffer_next_byte, mulle_buffer_peek_byte, and mulle_buffer_seek_byte. These work on any buffer state (including read-only) and provide safe, bounds-checked byte access for delimiter searching or protocol parsing.

Step-by-Step Implementation Patterns

Chunked Reading for Unknown File Sizes

When file statistics are unavailable or unreliable, read iteratively using guaranteed capacity:

#include <fcntl.h>
#include <unistd.h>
#include "mulle-buffer.h"

struct mulle_buffer buf = MULLE_BUFFER_DATA(NULL);  // start with default allocator

int fd = open("data.bin", O_RDONLY);
if (fd < 0) return -1;

while (1) {
    void *dst = mulle_buffer_guarantee(&buf, 8192);  // reserve 8KB
    if (!dst) break;  // allocation failure
    
    ssize_t n = read(fd, dst, 8192);
    if (n < 0) { /* handle error */ break; }
    if (n == 0) break;  // EOF
    
    mulle_buffer_advance(&buf, (size_t)n);  // commit actual bytes read
}

mulle_buffer_set_readonly(&buf);  // protect completed data
close(fd);

This pattern minimizes reallocations by growing the buffer exponentially while reading fixed-size chunks.

Zero-Copy Memory Mapping

When the entire file fits in addressable memory, map it directly:

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include "mulle-buffer.h"

int fd = open("largefile.bin", O_RDONLY);
struct stat st;
fstat(fd, &st);

void *map = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
struct mulle_buffer buf = MULLE_BUFFER_STATIC_DATA(map, st.st_size, NULL);

mulle_buffer_set_readonly(&buf);  // protect the mapping

// Direct pointer access for parsing
const void *data = mulle_buffer_reference_bytes(&buf, mulle_buffer_get_length(&buf));
// process data...

munmap(map, st.st_size);
close(fd);

This approach bypasses the heap entirely for the file content, though the mapping itself consumes virtual address space.

Key Source Files and Implementation Details

According to the mulle-c/mulle-buffer source code, three headers provide the complete toolkit:

• src/mulle-buffer.h: Public API containing mulle_buffer_guarantee, mulle_buffer_advance, mulle_buffer_set_readonly, and mulle_buffer_reference_bytes. Contains inline wrappers for high-performance operations.

• src/mulle--buffer.h: Low-level implementation exposing internal _mulle__buffer_* functions used by the public inline wrappers. Useful for understanding allocation strategies.

• src/mulle-flushablebuffer.h: Defines struct mulle_flushablebuffer and the mulle_flushablebuffer_flusher_t callback type for implementing custom streaming readers.

Summary

• Use mulle_buffer_guarantee to reserve writable capacity before system calls, eliminating temporary buffers.
• Commit bytes with mulle_buffer_advance after read(2) returns to maintain accurate length tracking.
• Wrap mmap regions using MULLE_BUFFER_STATIC_DATA for true zero-copy file access without heap allocation.
• Lock completed buffers with mulle_buffer_set_readonly to prevent accidental writes and enable safe sharing.
• Extract raw pointers via mulle_buffer_reference_bytes for parser integration without data duplication.
• Implement streaming through mulle_flushablebuffer in src/mulle-flushablebuffer.h when processing files larger than available RAM.

Frequently Asked Questions

Does mulle-buffer provide a built-in function to read files directly?

No. The library intentionally operates as a low-level abstraction in src/mulle-buffer.h. You implement efficient file reading by combining mulle_buffer_guarantee with POSIX read(2) or fread, or by wrapping memory-mapped regions with MULLE_BUFFER_STATIC_DATA. This design keeps the core library agnostic to I/O mechanisms while supporting optimal performance.

How do I prevent buffer reallocations while reading large files?

Call mulle_buffer_guarantee with your desired chunk size (e.g., 64 KB) before each read operation. If the existing capacity suffices, the function returns the current write pointer without reallocating. Only when the internal storage fills does the buffer grow, typically doubling in size to maintain amortized O(1) append complexity.

Can I use mulle-buffer with memory-mapped files safely?

Yes. Initialize the buffer with MULLE_BUFFER_STATIC_DATA pointing to your mmap region, then immediately call mulle_buffer_set_readonly. This prevents write attempts that would trigger segmentation faults on read-only mappings. The buffer provides full read API access—including mulle_buffer_next_byte and mulle_buffer_seek_byte—without copying data from the mapped region.

What is the difference between mulle_buffer and mulle_flushablebuffer?

struct mulle_buffer in src/mulle-buffer.h is the base growable-memory container. struct mulle_flushablebuffer in src/mulle-flushablebuffer.h extends this with a callback hook (mulle_flushablebuffer_flusher_t) that triggers when the buffer fills. Wire this flusher to fread or custom read functions to implement streaming readers that process files chunk-by-chunk without loading the entire contents into memory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}