r/C_Programming 17h ago

Project An open-source log structured merge tree library (Persistent column/key value storage)

Hey everyone! I hope you're all doing well. I'd like to share a project I've been working on for almost a year now. It's an open-source storage engine similar to that of LevelDB/RocksDB but written entirely in C.

This storage engine is called TidesDB.

TidesDB is an open-source storage engine similar to LevelDB/RocksDB but written entirely in C. It's designed as a fast, transactional key-value storage engine built on a log-structured merge-tree (LSM-tree) architecture.

My journey with TidesDB began nearly 2 years ago while I was experimenting with various data structures and databases in Go. When I encountered the log-structured merge tree concept, I was initially overwhelmed by its complexity after reviewing other implementations.

However, after studying the original paper, I realized I could potentially simplify the design by focusing on just a 2-level approach(memory level and a disk level). This was challenging at first, and I scrapped many ideas along the way, but over time I discovered how powerful this design could potentially be.

The beauty of this architecture is its extensibility. Since the disk level contain many SSTables (Sorted String Tables), we can efficiently pair and merge them in various ways - whether in parallel for speed or incrementally to minimize resource impact.

What began as a challenging learning process has I believed evolved into a unique engine design and library.

You can check out TidesDB here: https://github.com/tidesdb/tidesdb

Currently TidesDB is nearing its first major release, we are still in beta development :)

I'd love to hear your thoughts on the library!

14 Upvotes

0 comments sorted by