Show HN: EloqKV – Scalable distributed ACID key-value database with Redis API

> EloqKV brings significant innovations to database design

What is the novel part? I read your "Introduction to Data Substrate" blog article and the architecture you are describing sounds like NuoDB from the early 2010s. The only difference is that NuoDB scales out the in-memory cache by adding more of what they call "Transaction Engine" nodes whereas you are scaling up the "TxMap" node?

See also Viktor Leis' CIDR 2023 paper with the Great Phil Bernstein:

* https://www.cidrdb.org/cidr2023/papers/p50-ziegler.pdf

* https://youtu.be/tiMvcqIfWyA

Redis's transaction api is terrible, and doesn't work across shards. Any reasonable transactions are done in Lua, and because Lua mostly works well, there's not a lot of pressure to fix transactions.

However, if you're giving redis access to different tenants, Lua is too dangerous.

I'd love to see a "real" transaction API for Redis.

This is really interesting! The performance looks impressive. I’ve always struggled with the Redis/MySQL two-tier architecture because they are two completely different systems. Porting an application is always tricky due to API incompatibility, and maintaining consistency between the two is a huge hassle. If there’s any issue on the cache side (like performance jitters or a few servers going down), the DBMS part can quickly get overwhelmed. This kind of cascading failure is a common problem in distributed systems. There are some great discussions on this topic over at https://danluu.com/cache-incidents/. BTW, although the formatting is a bit rough, it's a great read for anyone interested in caching. Back to the topic, I always hoped that new advancements in database systems can alleviate or eliminate this problem, your work seems to be on the right track.

Jepsen test?

It's interesting that you have a Cassandra version. I usually think of Cassandra as a standalone database. Why would you layer one database on top of another?

Show us the Jepsen tests. Hire Aphyr to test your database.

It does seem that there are a lot of activities to replace/enhance/improve Redis lately. Dragonfly, Garnet from Microsoft, and the BSD licensed Redis fork Valkey, to name just a few. Now this.

What is license ? I see Download but it is not clear if it is Open Source

Note that this binary release is a preview of EloqKV. We're actively developing a cloud-native version with seamless scalability and a serverless experience. Stay tuned—it's coming soon!

Is the WAL 1:1 with a TxMap instance? Is the TxMap 1:1 with a node? For a distributed transaction with distributed storage and multiple nodes how does a transaction get coordinated?

Yet another distributed database announcement with nary a mention of CAP tradeoffs. Then again, no sales pitch starts with the limitations. Also, one man's ACID isn't another person's. At least they aren't advertising SQL joins with handwaves to the distributed complexities.

It superficially sounds like a series of server processes fronting actual database servers, which sounds like another layer of partition vulnerability and points of failure. But I also had similar high level concerns about the complexity of FoundationDB, people seem satisfied as to the validity of that architecture.

I fail to see how one would scale underlying resources if the persistence is done by various storage systems, and you'd be subject to limitations of those persistence engines. That sounds like a suspect claim, like the "Data Substrate" is a big distributed cache that scales in front of delayed persistence to an actual database. Again, sounds like oodles of opportunities for failure.

"Data substrate draws inspirations from the canonical design of single-node relational database management systems (RDBMS)." Look, I don't think a good distributed database starts from a CA system and bolts on partition tolerance. I get you can get far with big nodes, fast networks, and sharding but ... I mean, they talk about B+ trees but a foundational aspect of Cassandra and other distributed databases is that B+ trees don't scale past a certain point, because there exist trees so deep/tall that even modern hardware chokes on it, and updating the B+ tree with new data gets harder as it gets bigger.

As others have said, I'll leave it to Aphyr to possibly explain.

This binary release is a preview of EloqKV. We're actively developing a cloud-native version with seamless scalability and a serverless experience. Stay tuned—it's coming soon!

License?