Korvus: Single-Query RAG with Postgres

Hey fellow open-source enthusiasts,

We built Korvus, an open-source RAG (Retrieval-Augmented Generation) pipeline that consolidates the entire RAG workflow - from embedding generation to text generation - into a single SQL query, significantly reducing architectural complexity and latency.

Here's some of the highlights:

- Full RAG pipeline (embedding generation, vector search, reranking, and text generation) in one SQL query

- SDKs for Python, JavaScript, and Rust (more languages planned)

- Built on PostgreSQL, leveraging pgvector and pgml

- Open-source, with support for open models

- Designed for high performance and scalability

Korvus utilizes Postgres' advanced features to perform complex RAG operations natively within the database. We're also the developers of PostgresML, so we're big advocates of in-database machine learning. This approach eliminates the need for external services and API calls, potentially reducing latency by orders of magnitude compared to traditional microservice architectures. It's how our founding team built and scaled the ML platform at Instacart.

We're eager to get feedback from the community and welcome contributions. Check out our GitHub repo for more details, and feel free to hit us up in our Discord!

Very cool. I see more languages planned in your comment. Are you looking for community help developing SDKs in other languages? After spending an entire Saturday running a RAG pipeline for a POC for a "fun" side project, I definitely would've loved to have been able to use this instead.

I spent too long reading Python docs because I haven't touched the language since 2019. Happy to help develop a Ruby SDK!

Does this work my running LLM such as Llama directly on the database server? If so, does that mean that your database and the LLM are competing for the same CPU and memory resources?

Can it run the LLM on a GPU?

I'm not sure if this is a good idea, just like pretending a network request is a function call, it hides a lot of elements that shouldn't be ignored. I still prefer to clearly explicit embedding, LLM generation, etc.

As a long time user of pgvector I'm really hyped up about this. Korvus has the potential to reduce a lot of the repetitive code in projects I work on.

You mention pulling models from huggingface for document embedding. Is it possible to pass an hf token to use private models?

I train domain and language-specific[0] embedding and conversational models and if I can use them in Korvus I'll most likely switch to it overnight.

[0]: https://sawalni.com/developers

This sounds very promising, but let me ask an honest question: to me, it seems like databases are the hardest part to scale in your average IT infrastructure. How much work does it add to the database if you let it make all the ML related work as well? How much work is saved by reducing the number of necessary queries?

I was expecting to see something like a foreign table that managed the upload, chunking, embedding, everything in a transparent manner. But what I found in the examples was some Python code that look a lot like what the other frameworks are doing.

What am I missing? Honest question. I want to likes this :)

Is there any way to deploy this to an existing postgres database or does it need to use the docker instance.

This looks exciting! Will definitely be testing it out in the coming days.

I see you offer re-ranking using local models, will there be build-in support for making re-ranking calls to external services such as cohere in the future?

You emphasize single-query, but I can't find the query. Where can I see it?

Interesting! Is there a way to deploy this on AWS RDS?

What LLM system does it use to run models? Does it support ollama?

This looks great, thanks! After being disappointed by how flaky gpt-4-turbo's RAG is, I want to set up my own, so this came at the right time.

One question: Can I use an external model (ie get the raw RAG snippets, or prompt text)? Or does it have to be the one specified in Korvus?

Is RAG the new DAG? outoftheloop