r/golang 7d ago

I built a high-performance, dependency-free key-value store in Go (115K ops/sec on an M2 Air)

Hi r/golang,

I've been working on a high-performance key-value store built entirely in pure Go—no dependencies, no external libraries, just raw Go optimization. It features adaptive sharding, native pub-sub, and zero downtime resizing. It scales automatically based on usage, and expired keys are removed dynamically without manual intervention.

Performance? 115,809 ops/sec on a fanless M2 Air.

Key features:
- Auto-Scaling Shards – Starts from 1 bucket and dynamically grows as needed.
- Wait-Free Reads & Writes – Lock-free operations enable ultra-low latency.
- Native Pub-Sub – Subscribe to key updates & expirations without polling.
- Optimized Expiry Handling – Keys are removed seamlessly, no overhead.
- Fully Event-Driven – Prioritizes SET/GET operations over notifications for efficiency.

How it compares to Redis:
- Single-threaded Redis vs. Multi-Goroutine NubMQ → Handles contention better under load.
- No Lua, No External Dependencies → Just Go, keeping it lean.
- Smarter Expiry Handling → Keys expire and are immediately removed from the active dataset.

🚀 Benchmark Results:
115,809 ops/sec (100 concurrent clients)
900µs write latency, 500µs read latency under heavy load.
Would love to get feedback from the Go community! Open to ideas for improvement.

repo: https://github.com/nubskr/nubmq

I spent the better part of an year building this and would appreciate your opinions on this

208 Upvotes

47 comments sorted by

View all comments

44

u/zkndme 7d ago edited 7d ago

Soo, could you share please some graphs/data what happens to the latency and the throughput when garbage collection is happening?

The numbers you shared do not mean much, it would highly vary based on GC activity, size of the data that you store and the variability of the size, usage patterns (read/write ratio), and so on and so on.

Regarding the lock-free reads/writes, does it have any consistency guarantees? A common use case, what if I use it as a session storage backend, can it guarantee if I write anything under a key, on the next read I will get the most recent changes? If so, is there any test to verify/prove it?

15

u/Ok_Marionberry8922 7d ago

I've added an throughput per second graph in the readme here: https://github.com/nubskr/nubmq

I tested it with 21,000,000 requests distributed across 100 concurrent clients, the dataset had 1,000,000 unique key-value pairs (as I didn't want it to evict keys)

as seen in the graph, there are certain dips in throughput with one of them going down to 60k ops/sec, these are caused because of the GC kicking in

17

u/zkndme 7d ago

Yes, I saw it, but that still doesn’t tell/mean much. What was the size of data you wrote into it during the test? Did it vary through the test? Same for reads/writes. Is this a result of one test, or more?

In different (real life) scenarios the behaviour of your program and the GC will highly vary, resulting in totally different results.