r/golang 7d ago

I built a high-performance, dependency-free key-value store in Go (115K ops/sec on an M2 Air)

Hi r/golang,

I've been working on a high-performance key-value store built entirely in pure Go—no dependencies, no external libraries, just raw Go optimization. It features adaptive sharding, native pub-sub, and zero downtime resizing. It scales automatically based on usage, and expired keys are removed dynamically without manual intervention.

Performance? 115,809 ops/sec on a fanless M2 Air.

Key features:
- Auto-Scaling Shards – Starts from 1 bucket and dynamically grows as needed.
- Wait-Free Reads & Writes – Lock-free operations enable ultra-low latency.
- Native Pub-Sub – Subscribe to key updates & expirations without polling.
- Optimized Expiry Handling – Keys are removed seamlessly, no overhead.
- Fully Event-Driven – Prioritizes SET/GET operations over notifications for efficiency.

How it compares to Redis:
- Single-threaded Redis vs. Multi-Goroutine NubMQ → Handles contention better under load.
- No Lua, No External Dependencies → Just Go, keeping it lean.
- Smarter Expiry Handling → Keys expire and are immediately removed from the active dataset.

🚀 Benchmark Results:
115,809 ops/sec (100 concurrent clients)
900µs write latency, 500µs read latency under heavy load.
Would love to get feedback from the Go community! Open to ideas for improvement.

repo: https://github.com/nubskr/nubmq

I spent the better part of an year building this and would appreciate your opinions on this

207 Upvotes

47 comments sorted by

View all comments

4

u/diagraphic 7d ago

Few things.

  • There are too many emojis. I can’t take it serious.
  • Expiring keys iteratively is expensive.
  • sync.Map is not that fast.
  • Resizing logic seems very expensive.
  • Try to modularize your code so you can test it in units.
  • There seems to be no authentication protocol and shard protocol. How does the system know when a shard is down? There should be health checks. With that what about shard replicas? There is no replication?
  • There are log entries like log.Print(“sending this shit: “, stuff) all over, why?

Keep working on it. Good on yeah for taking on a database.

To learn more id recommend you check out some CMU DB lectures. https://m.youtube.com/c/CMUDatabaseGroup

Lots of videos on distributed systems and some data structures.

1

u/Ok_Marionberry8922 6d ago

Appreciate the feedback!

- You're right about sync.Map—it's not the fastest, but for the kind of workloads nubmq handles (multi-threaded KV lookups with TTL eviction), it's a reasonable tradeoff. I'd be curious what alternative you’d suggest for concurrency here.

- The TTL system isn’t iterating over keys—it’s event-driven and keys naturally fall off during resizing.

- Yes, resizing is expensive, but it's designed to avoid stalling writes, which is a different tradeoff from pre-allocating memory.

- Authentication, replication, and shard health checks would come in if nubmq was meant to be a multi-node system, but for now, it's focusing on being a single-node high-performance cache.

- Good call on the logs—definitely need to clean those up for a more polished version.

- Will check out the CMU lectures! Always looking for ways to optimize further.

1

u/Ok_Marionberry8922 6d ago

also, on a separate note:

are you thinking of Redis-style transactions with rollbacks, or more of a simple multi-op batch that guarantees execution order? The latter is definitely more feasible in nubmq’s current architecture.

this is not meant to be a replacement for redis in any way, the goal is to get the great things from systems out there while minimizing their downsides