Understanding Croc's TCP Buffer Size and Transfer Chunking

Croc uses a fixed 64 KB TCP buffer size defined by models.TCP_BUFFER_SIZE to balance memory efficiency and network throughput, reading and writing data in uniform chunks via the chanFromConn and pipe functions in src/tcp/tcp.go.

Croc is a command-line tool for secure peer-to-peer file transfers that handles large payloads by streaming data over TCP in manageable pieces. Understanding how the schollz/croc repository implements TCP buffer sizing and transfer chunking is essential for optimizing transfer performance on different network conditions. This analysis examines the default 64 KB buffer constant, the low-level socket handling code, and how file data is segmented before hitting the wire.

Where the 64 KB Buffer Size is Defined

The foundation of croc's transfer logic rests on a single constant defined in src/models/constants.go. The variable models.TCP_BUFFER_SIZE sets the byte length for every read and write operation across the codebase.

By default, this value is initialized to 64 KB (64 * 1024 bytes). This constant appears in multiple critical paths, ensuring that memory allocation remains predictable and that both sender and receiver agree on the maximum payload size for each network round-trip.

How Croc Reads Data from TCP Connections

Incoming data is handled by the chanFromConn function located in src/tcp/tcp.go (lines 66–73). This function creates a buffered channel and spawns a goroutine that continuously reads from the TCP socket.

b := make([]byte, models.TCP_BUFFER_SIZE)
n, err := conn.Read(b)

The function allocates a byte slice of exactly TCP_BUFFER_SIZE bytes. When conn.Read(b) returns, it captures the actual number of bytes received (n) and copies only that segment into a new slice before sending it over the channel. This defensive copy prevents the underlying buffer from being overwritten by subsequent read operations while the data is in transit through the pipeline.

Writing and Forwarding Chunks with the Pipe Function

Once data enters the channel system, the pipe function (also in src/tcp/tcp.go, lines 88–105) manages full-duplex communication between two connections. This function receives byte slices from the channels and immediately writes them to the opposite socket.

// Simplified logic from the pipe implementation
conn2.Write(b1)

Because chanFromConn ensures each slice contains exactly the bytes received (up to the 64 KB limit), the pipe function can write the entire slice without additional chunking logic. This design keeps the write path simple and eliminates unnecessary memory copying or reassembly overhead.

File Chunking on the Sender Side

Before data ever reaches the TCP stack, croc prepares it by reading source files in identically sized chunks. In src/utils/utils.go (lines 387–393), the file-reading logic uses a buffer matching chunkSize, which defaults to the same value as models.TCP_BUFFER_SIZE.

This symmetry ensures that the amount of data placed on the network matches the receiver's read buffer exactly. When the sender reads 64 KB from disk, the receiver expects to consume 64 KB from its socket, creating a predictable flow-control rhythm that minimizes latency spikes.

Why 64 KB? Performance and Compatibility Trade-offs

The default buffer size represents a carefully chosen middle ground between several competing factors:

• Ethernet Efficiency: A 64 KB payload aligns well with standard Ethernet MTU multiples, reducing the number of system calls required to move large files while avoiding excessive memory pressure.
• Network Compatibility: Most operating systems, routers, and middleboxes handle 64 KB TCP segments without fragmentation or throttling, ensuring reliable transfers across diverse network topologies.
• Deterministic Flow Control: Using a fixed constant simplifies the handling of partial reads, timeouts, and congestion events in chanFromConn, making the codebase easier to maintain and debug.

Customizing the Buffer Size

While 64 KB suits most scenarios, high-latency or high-bandwidth links may benefit from larger buffers to improve throughput. Croc exposes this tuning via the --buffer command-line flag, parsed in src/tcp/options.go. When specified, this flag updates the runtime value of models.TCP_BUFFER_SIZE before any connections are established.

Increasing the buffer size can reduce the overhead of system calls on long-distance links, though it trades higher memory usage per connection for potentially faster transfer speeds.

Practical Code Examples

Sending a File in 64 KB Chunks

This example demonstrates how to manually read a file and transmit it using croc's encryption and chunking logic:

c, _, _, err := croc.ConnectToTCPServer("example.com:9000", "", "")
if err != nil { 
    log.Fatal(err) 
}

file, _ := os.Open("bigfile.bin")
defer file.Close()

buf := make([]byte, croc.TCP_BUFFER_SIZE) // 64 KB
for {
    n, err := file.Read(buf)
    if n > 0 {
        // Encrypt and send; crypt package handles optional encryption
        enc, _ := croc.Encrypt(buf[:n])
        c.Send(enc)
    }
    if err == io.EOF { 
        break 
    }
    if err != nil { 
        log.Fatal(err) 
    }
}
c.Close()

Receiving Incoming Chunks

On the receiving side, data arrives pre-segmented into the buffer size used by the sender:

for {
    data, err := c.Receive()
    if err != nil { 
        break 
    }
    // Data contains exactly one 64 KB chunk (or smaller for the final piece)
    process(data)
}

Summary

• Default Size: Croc uses a 64 KB (64*1024) buffer defined by models.TCP_BUFFER_SIZE in src/models/constants.go.
• Socket Reading: The chanFromConn function in src/tcp/tcp.go allocates buffers of this size and copies only valid bytes to prevent overwrites.
• Socket Writing: The pipe function writes received slices directly to peers without additional chunking, relying on the fixed size for flow control.
• File I/O: src/utils/utils.go reads source files using the same chunk size to maintain symmetry between disk and network operations.
• Customization: Users can override the default via the --buffer flag parsed in src/tcp/options.go to optimize for specific network conditions.

Frequently Asked Questions

What is the default TCP buffer size in croc?

Croc defaults to a 64 KB buffer size for all TCP read and write operations. This value is defined by the constant models.TCP_BUFFER_SIZE in src/models/constants.go and is used consistently across the file transfer pipeline to balance memory usage and network efficiency.

Where is the buffer size defined in the croc source code?

The buffer size is defined as TCP_BUFFER_SIZE in src/models/constants.go. This constant is referenced by the TCP handling logic in src/tcp/tcp.go and the file utilities in src/utils/utils.go, ensuring uniform chunking throughout the application.

How does croc handle partial reads from the TCP socket?

In the chanFromConn function within src/tcp/tcp.go, croc allocates a full 64 KB buffer but only copies the actual bytes received (n) into a new slice before sending it over the channel. This approach prevents data corruption from subsequent read operations while allowing the system to handle varying network speeds gracefully.

Can I increase the buffer size for faster transfers on high-latency networks?

Yes. Croc provides the --buffer command-line option (parsed in src/tcp/options.go) that allows you to override the default 64 KB value at runtime. Increasing this value can improve throughput on high-bandwidth, high-latency links by reducing the frequency of system calls, though it will increase memory consumption per active connection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}