Show HN: Drop-In SQS Replacement Based on SQLite

The whole idea here is superlative.

+1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost.

I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge.

Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong.

BTLE sensors are perfectly happy talking to my Apple Watch directly with enough debugging.

I'd argue the trip through cloud was not a win and should be corrected in the next generation of tools like this, where mobile is already well-primed for SQLite.

Keeping questions from scale & benchmarks aside, this is a cool thing for functional/unit testing module that uses SQS, instead of dumb mocks.

Tangential but I looked at the codebase - because I like to imagine I can code (I can’t) - and for a laymen, I could follow the code.

Makes me think quite positively of go lang and of the dev for designing in such a way. Can understand why teams like it because of easier maintenance

Quite elegant from my uneducated pov.

> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.

are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data

The goals are a little different but I think it's worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. https://github.com/softwaremill/elasticmq

This is super cool! I love projects that aim to create simple self-hostable alternatives to popular services.

I assume this would work without much issue with Litestream, though I'm curious if you've already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.

Congratulations!

I also love writing AWS API-compatible services. That's why I did Dyna53 [1] ;P

(I know, unrelated, but hopefully funny)

[1] https://dyna53.io

One quick suggestion on project structure:

Move all the structs from models/ into the root directory.

This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.

Correct me if I’m wrong but: SQLite sounds like: runs on one server. While that will work most of the time, it won’t work 100% of the time. I don’t know the specifics, but I’m fairly sure if a queue server crashes, SQS will keep working, as stuff is redundant. So while it can work in a best case, this (probably) won’t have the same reliability as SQS has..

Good, we need open implementations of all the AWS stuff.

I swear they reimplement stuff we have just so there are more places to bill us.

Why not use LocalStack? It has SQS and a lot of AWS services for testing/development. Well documented, open source.

https://docs.localstack.cloud/overview/

I may be asking a naive question, but what is the rationale behind disabling foreign key support and using them anyway in the database schema? See https://github.com/poundifdef/SmoothMQ/blob/46f8b22/queue/sq...

The "TODO: check for errors" comment, combined with what seems like disabling foreign key constraint checks, makes me a bit hesitant to try this out.

Could you elaborate why you didn't choose e.g. RabbitMQ? I mean you talk about AMQP and such, it seems to me, that a client-side abstraction would be much more efficient in providing an exit strategy for SQS than creating an SQS "compatible" broker. For example in the Java ecosystem there is https://smallrye.io/smallrye-reactive-messaging/latest/ that serves a similar purpose.

I wrote sasquatch which is similar. sasquatch is more simple than SQS and queues are virtual (i.e the queue exists because you put a message in it, and is gone when there's no messages in it).

https://github.com/crowdwave/sasquatch

sasquatch is also a message queue, also written in Golang and also based on sqlite.

sasquatch implements behaviour very similar to SQS but does not attempt to be a dropin replacement.

sqsquatch is not a complete project though, nor even really a prototype, just early code. Likely it does not compile.

HOWEVER - sasquatch is MIT license (versus this project which is AGPL) so you are free to do with it as you choose.

sasquatch is a single file of 700 lines so easy to get your head around: https://raw.githubusercontent.com/crowdwave/sasquatch/main/s...

Just remember as I say it's early code, won't even compile yet but functionally should be complete.

So I assume it does the back-end as well?

I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers?

As an SQS user, a table of SQS features vs SmoothMQ would be handy. If it's just an API-compatible front-end then that would be good to know. But if it does more that would also be good to know.

The reason you'd use this is because there are lots of clients who still want on-prem solutions (go figure). Being able to switch targets this way would be handy.

Very cool. I needed something like this. Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics. Docs for current features are more important though.

Perhaps do a small example application.

  go  get github.com/poundifdef/SmoothMQ/models
  go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 requires go >= 1.22.2; switching to go1.22.4
  go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 (matching github.com/poundifdef/SmoothMQ/models@upgrade) requires github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60: parsing go.mod:
          module declares its path as: q
                  but was required as: github.com/poundifdef/SmoothMQ

This is very interesting! The self-hosted aspect is something I'll have to consider for certain purposes.

My lab also developed an SQS-esque system based on the filesystem, so no dependencies whatsoever and no need for any operational system other than the OS. It doesn't support all SQS commands (because we haven't needed them), but it also supports commands that SQS doesn't have (like release all messages to visible status).

https://github.com/seung-lab/python-task-queue

I think it’s better to say it’s sqs compatible versus an sqs replacement.

I'd have named it MQLite. Congrats on finishing and delivering a project, this is quite a challenge in itself. I think SQLite can be a great alternative for many kinds of projects.

This looks great! How were you planning on tackling distributed?

There is a lot of new ecosystem being built around SQLite in the browser using wasm, as a primary data store or as a local replica for every client, and there are some interesting interactions with crdt and peer to peer applications; does this fit into your business case? Would be interesting to see a massively distributed - via browser embedding - queue, that even uses standard sqs bindings.

This is really pedantic, totally outside any actual functionality, and may be a me thing, but a trigger for me is seeing left-justified (or worse, centered), columns of numbers (eg: your number of messages column).

I'll grant it's small to infinitesimal, but you asked for feedback.

Minor thing - the gif browser demo should have been maximized. The video zooming into the action on what would have fit onto a single screen was distracting.

Are there maintenance actions that the admin needs to perform on the database? How are those done?

A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.

How would one use this to replace Sidekiq for instance?

Rate limiting is something I miss on Sidekiq (only available on the premium plan that I can't afford) and the gems that extend it break compatibility often.

I was thinking of using Redis Stream instead of SQS, what would be better Redis or SmoothMQ?

Glad to find out what you've been up to! Very cool and I am excited to see where you take this.

Actually pretty excited to try this, there are so many cases where I need a bare-bones (aka "simple"), local, persistent queue, but all the usual suspects (amqp, apache whatever, cloud nonsense, etc.) are way too heavy.

I'll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?

Not sure about the goal of providing a hosted service cheaper than SQS. SQS is already one of the cheapest services on Earth. It's pretty hard to spend more than a few bucks a month, even if you really try!

sounds like a great project. How do you handle performance with large message sizes?

Thanks!

Loving all these self-hosted KISS stuff :)

I love that we're seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I'm hopeful the next generation will have their introduction to building apps with simple, easy to manage & deploy stacks. The more I learn about SQLite, the more I love it.

SQS: Amazon Simple Queue Service

The only reason I use SQS is because how it's compatible with other AWS Services, I can have an S3 bucket whenever there's an upload I can send an SQS Message and I can have it trigger a lambda.