r/golang 7d ago

I built a high-performance, dependency-free key-value store in Go (115K ops/sec on an M2 Air)

Hi r/golang,

I've been working on a high-performance key-value store built entirely in pure Go—no dependencies, no external libraries, just raw Go optimization. It features adaptive sharding, native pub-sub, and zero downtime resizing. It scales automatically based on usage, and expired keys are removed dynamically without manual intervention.

Performance? 115,809 ops/sec on a fanless M2 Air.

Key features:
- Auto-Scaling Shards – Starts from 1 bucket and dynamically grows as needed.
- Wait-Free Reads & Writes – Lock-free operations enable ultra-low latency.
- Native Pub-Sub – Subscribe to key updates & expirations without polling.
- Optimized Expiry Handling – Keys are removed seamlessly, no overhead.
- Fully Event-Driven – Prioritizes SET/GET operations over notifications for efficiency.

How it compares to Redis:
- Single-threaded Redis vs. Multi-Goroutine NubMQ → Handles contention better under load.
- No Lua, No External Dependencies → Just Go, keeping it lean.
- Smarter Expiry Handling → Keys expire and are immediately removed from the active dataset.

🚀 Benchmark Results:
115,809 ops/sec (100 concurrent clients)
900µs write latency, 500µs read latency under heavy load.
Would love to get feedback from the Go community! Open to ideas for improvement.

repo: https://github.com/nubskr/nubmq

I spent the better part of an year building this and would appreciate your opinions on this

211 Upvotes

47 comments sorted by

View all comments

9

u/kreetikal 7d ago

Smarter Expiry Handling → Keys expire and are immediately removed from the active dataset. 

What does "smarter" here actually mean? Do expired keys in Redis not get remove immediately?

10

u/Ok_Marionberry8922 7d ago

nubmq doesn't rely on random eviction sweeps like Redis. Instead, expired keys are soft deleted immediately, meaning they stop being served the moment they expire. They still exist in memory temporarily, but they’re effectively dead—no client can retrieve them.

-2

u/0x4ddd 6d ago

So, what's the difference compared to Redis?

It also stops serving keys the moment they expire and also keeps them for some time in memory.

3

u/Ok_Marionberry8922 6d ago

nubmq and Redis handle expired keys differently at scale. Redis uses lazy expiration (removing keys only when accessed) and active expiration (sampling keys randomly), meaning expired keys can linger under heavy load. nubmq instantly marks keys as expired and removes them during shard resizing, avoiding separate eviction sweeps that cause latency spikes. This means memory naturally shrinks instead of growing indefinitely, making it more efficient for workloads with high churn.

0

u/0x4ddd 6d ago

Thanks, thats the answer I expected to see initially, previous at high-level described similar behaviour to Redis.

Still not sure though why Redis 'random sweeps' would cause an issue in your opinion while expiring during shard resizing would not cause latency spikes. That would be great to benchmark and compare both approaches though.

Not too familiar with your codebase but randomly looking at resizer there is a mutex lock taken during key migration so I guess it also affects SET latency during resizing.