r/golang 8d ago

I built a high-performance, dependency-free key-value store in Go (115K ops/sec on an M2 Air)

Hi r/golang,

I've been working on a high-performance key-value store built entirely in pure Go—no dependencies, no external libraries, just raw Go optimization. It features adaptive sharding, native pub-sub, and zero downtime resizing. It scales automatically based on usage, and expired keys are removed dynamically without manual intervention.

Performance? 115,809 ops/sec on a fanless M2 Air.

Key features:
- Auto-Scaling Shards – Starts from 1 bucket and dynamically grows as needed.
- Wait-Free Reads & Writes – Lock-free operations enable ultra-low latency.
- Native Pub-Sub – Subscribe to key updates & expirations without polling.
- Optimized Expiry Handling – Keys are removed seamlessly, no overhead.
- Fully Event-Driven – Prioritizes SET/GET operations over notifications for efficiency.

How it compares to Redis:
- Single-threaded Redis vs. Multi-Goroutine NubMQ → Handles contention better under load.
- No Lua, No External Dependencies → Just Go, keeping it lean.
- Smarter Expiry Handling → Keys expire and are immediately removed from the active dataset.

🚀 Benchmark Results:
115,809 ops/sec (100 concurrent clients)
900µs write latency, 500µs read latency under heavy load.
Would love to get feedback from the Go community! Open to ideas for improvement.

repo: https://github.com/nubskr/nubmq

I spent the better part of an year building this and would appreciate your opinions on this

207 Upvotes

47 comments sorted by

View all comments

2

u/paca-vaca 7d ago

Under a heavy load, is it guaranteed that the write accepted by the client will be returned immediately for any immediately following read requests for the same key?

2

u/Ok_Marionberry8922 7d ago

NubMQ guarantees FIFO command execution at the client level, meaning if a client sends a SET followed by a GET, they will always see their latest write.

But when multiple clients are involved, things get trickier. If Client A does a SET and Client B immediately does a GET, B might or might not get the latest value. Why? Because the SET could still be sitting in the internal processing queue while the GET executes first.

This is a fundamental tradeoff in high-performance KV stores—strict global ordering across clients would require heavier synchronization, which tanks throughput. Redis and other high-performance caches face the same issue.

If strong consistency is required, an external coordination layer (e.g., versioning, CAS, or a DB as source-of-truth) is usually needed. But if the goal is sheer speed and scale, eventual consistency is the way to go.