Building a Kafka Clone from Scratch in Go

Why would anyone build a Kafka clone in Go from scratch?

In an era of managed services and high-level abstractions, I’ve found that we often treat our infrastructure like a black box. We send a message to a topic, and it “just works.” But as a systems engineer, that “just works” is an invitation to curiosity. To truly understand a distributed system, I believe you have to feel the friction of its gears. You have to handle the bytes, manage the file descriptors, and worry about the race conditions yourself.

This is the story of Samsa—a lightweight, high-performance Kafka clone I built in Go. I didn’t build it to replace the massive JVM-based clusters that power the modern web; I built it as an educational odyssey into the heart of bit-flipping, disk persistence, and network protocols.

🏗️ Architecture: The Big Picture

When I set out to build Samsa, I chose a Hexagonal (Clean) Architecture. I wanted the core business logic—the “how” of a message broker—to be entirely decoupled from the “where” (the network or the disk).

The Request Lifecycle

At the center of Samsa is a request pipeline that I designed to feel like a well-oiled assembly line:

Network Layer: A concurrent TCP listener accepts connections and hands them off to a pool of goroutines.
Protocol Engine: This is the “brain.” It reads raw bytes from the wire, identifies the API Key (Produce, Fetch, etc.), and decodes the Kafka-compatible binary format into structured Go structs.
Handlers: These are the orchestrators. My Produce handler doesn’t know about TCP; it only knows how to validate a request and ask the Storage Engine to persist it.
Storage Engine: This is the “source of truth.” It manages an append-only log on disk and an in-memory index for rapid lookups.
The Return: The result is wrapped back into a binary response, encoded, and sent back across the TCP socket.

This flow ensures that if I ever wanted to swap out the disk for S3 or the TCP listener for QUIC, the core logic of Samsa would remain untouched.

🛠️ The Hard Parts: Engineering the Core

Building a broker isn’t about writing a lot of code; it’s about writing the right code for cases where performance and correctness are non-negotiable.

Hard Part 1: Demystifying the Binary Protocol

Kafka doesn’t use JSON. It doesn’t use Protobuf. It uses a bespoke, high-performance binary protocol. To speak Kafka, I had to learn to speak “Big Endian” and “Varints.”

I made a conscious decision to avoid reflection. In Go, reflection is powerful but slow and often hides bugs until runtime. Instead, I built a manual Reader and Writer utility. This choice forced me to account for every single byte. When I was decoding a RecordBatch v2, I wasn’t just mapping fields; I was navigating a specific memory layout.

// DecodeRecordBatch parses a Kafka Record Batch (v2) and returns individual records.
func DecodeRecordBatch(data []byte) ([]Record, error) {
    // ...
    baseOffset := int64(binary.BigEndian.Uint64(data[pos : pos+8]))
    pos += 8

    batchLength := int32(binary.BigEndian.Uint32(data[pos : pos+4]))
    pos += 4

    // Skip technical headers (CRC, Attributes, MaxTimestamp, etc.)
    pos += 4 + 1 + 4 + 2 + 4 + 8 + 8 + 8 + 2 + 4

    recordsCount := int32(binary.BigEndian.Uint32(data[pos : pos+4]))
    pos += 4
    
    // ... iterate and decode individual records ...
}

The “aha!” moment came when I realized how efficient this is. There’s no ceremony. You read 4 bytes, you have an integer. You read N bytes, you have a string. It’s direct, it’s fast, and it’s remarkably transparent.

Hard Part 2: High-Performance Disk I/O

A broker is only as good as its durability. I implemented an Append-Only Log strategy for Samsa. When a producer sends a message, I don’t look for an empty spot in a database; I simply gravitate to the end of the file and append the bytes. Sequential I/O is the “secret sauce” of Kafka’s throughput, and I wanted to replicate that here.

But writing to a file isn’t enough. If the power goes out, those bytes might still be sitting in the OS’s page cache. I had to ensure data integrity by explicitly calling fsync.

f, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
    return err
}
if _, err := f.Write(p.Records); err != nil {
    f.Close()
    return err
}

// The "Durability" Step: Ensure the OS flushes to physical media
if err := f.Sync(); err != nil {
    slog.Error("disk sync failure", slog.Any("error", err))
}
f.Close()

To complement the logs, I implemented an In-Memory Offset Index. Reading a 10GB log file to find offset #500,000 would be a disaster. My index maps offsets to byte positions, allowing the Fetch handler to jump straight to the correct location using f.ReadAt.

Hard Part 3: The Unified Binary & DX

I wanted Samsa to be a joy to use, not just to build. Using the cobra library, I merged the server and the administrative client into a single binary.

One of the subtler challenges I faced was Observability. When you run samsa server, you want machine-readable JSON logs for your ELK stack. But when you run samsa produce, you want pretty-printed text in your terminal. I solved this by leveraging the log/slog package, configuring handlers dynamically based on the specific command being executed.

# Server mode: JSON logs
$ samsa server
{"time":"...","level":"INFO","msg":"starting Kafka broker","addr":"0.0.0.0:9092"}

# CLI mode: Human-readable text
$ samsa topic create --name orders
INFO topic created successfully name=orders partitions=1

Hard Part 4: Building the Continuous Consume Loop

Making a stateless network request feel like a live stream (like tail -f) was a pure UX challenge for me. The Kafka Fetch API is request-response based. To create the samsa consume command, I had to implement a client-side polling loop that manages its own state:

Ask for messages starting at CurrentOffset.
If data comes back, print it and update CurrentOffset = LastOffset + 1.
If no data comes back, back off for 500ms and try again.

This “pull-based” approach is why Kafka scales so well—the broker doesn’t have to remember who is reading what; the client handles that state.

🛡️ Operational Safety: The Art of the Graceful Shutdown

In my experience, how you stop a system is just as important as how you start it. If you simply Ctrl+C a broker while it’s mid-write, you risk a corrupted log file.

I implemented a robust signal-trapping pattern in Samsa. When the application receives an interrupt, it stops accepting new connections, waits for in-flight requests to complete, and ensures all file handles are synced and closed properly.

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

// ... server logic ...

<-ctx.Done() // Wait for Ctrl+C
slog.Info("graceful shutdown initiated")
srv.Shutdown() // Flushes and closes everything

🏁 The Outcome: What I Built

Samsa is now a fully functional, cross-platform Kafka clone. It represents months of deep work and features:

Zero Dependencies: Just a single Go binary.
Durability: Persistence that respects the disk.
Observability: Structured logging for the modern dev.
Distribution: An automated CI/CD pipeline that ships binaries for Linux, macOS, and Windows.

It’s been a testament to the power of Go for systems programming. With just the standard library and a few high-quality packages like Cobra and Slog, I was able to build infrastructure that is both simple and powerful.

🌟 Join the Odyssey

I’ve made Samsa open source. It’s a playground for anyone who wants to learn about distributed systems, binary protocols, or high-performance Go.

Star the repo to show your support -> https://github.com/albertopastormr/samsa
Review the code and tell me where I can be more idiomatic.
Submit a PR: Want to implement log rolling? Compression? Multiple replicas? I’d love to see what you build.

Building Samsa reminded me that the “magic” in our infrastructure is just code—and that code is something we can all master, one byte at a time.

Built for education and performance.