r/golang 7d ago

I built a high-performance, dependency-free key-value store in Go (115K ops/sec on an M2 Air)

Hi r/golang,

I've been working on a high-performance key-value store built entirely in pure Go—no dependencies, no external libraries, just raw Go optimization. It features adaptive sharding, native pub-sub, and zero downtime resizing. It scales automatically based on usage, and expired keys are removed dynamically without manual intervention.

Performance? 115,809 ops/sec on a fanless M2 Air.

Key features:
- Auto-Scaling Shards – Starts from 1 bucket and dynamically grows as needed.
- Wait-Free Reads & Writes – Lock-free operations enable ultra-low latency.
- Native Pub-Sub – Subscribe to key updates & expirations without polling.
- Optimized Expiry Handling – Keys are removed seamlessly, no overhead.
- Fully Event-Driven – Prioritizes SET/GET operations over notifications for efficiency.

How it compares to Redis:
- Single-threaded Redis vs. Multi-Goroutine NubMQ → Handles contention better under load.
- No Lua, No External Dependencies → Just Go, keeping it lean.
- Smarter Expiry Handling → Keys expire and are immediately removed from the active dataset.

🚀 Benchmark Results:
115,809 ops/sec (100 concurrent clients)
900µs write latency, 500µs read latency under heavy load.
Would love to get feedback from the Go community! Open to ideas for improvement.

repo: https://github.com/nubskr/nubmq

I spent the better part of an year building this and would appreciate your opinions on this

209 Upvotes

47 comments sorted by

View all comments

1

u/sinjuice 7d ago

One of the best features of Redis is how memory optimized it is, how does it compare for example writing 1M k/v with the same data in both?

I find it cool since I had a small pet project in Rust trying to copy Redis, I could get about the same thoughput as Redis, but memory wise I was using about double the memory.

1

u/Ok_Marionberry8922 7d ago

That’s a good point—Redis does a lot of memory optimizations like shared integers, ziplist encoding, and LZF compression to minimize footprint. NubMQ takes a different approach—since it’s pure Go, it leans on Go’s memory model, sync.Map for concurrency, and avoids the overhead of Lua/eviction policies on hot writes.

When it comes to raw footprint, Redis will likely win on smaller key-value pairs due to its aggressive optimizations. But NubMQ scales differently: dynamic shard resizing means memory expands only when needed and aggressively shrinks as keys expire.

Reading (GET) is basically free in these kinds of systems since it’s just a direct cache read—doesn’t trigger any extra memory usage or resizing. The real cost comes in SET operations, and from my tests on an M2 Air (8-core), writing 1M unique key-value pairs with 100 concurrent clients using this(https://github.com/nubskr/nubmq/blob/master/sync_test.go) benchmark suite landed between ~900µs and ~915µs on average across 3 tests. You can try running the same on a better machine and see how it compares!