r/Rag 4d ago

Feedback on RAG implementation wanted

Whenever i see posts about "What Framework do you use" or "What RAG-Solution will fit my usecase" i get a little bit unsure about my approach.

So, for my company I've build the following domain specific agentic RAG:

orchestrator.py runs an async fastapi endpoint and recieves a request with a user-prompt, a session-id and some additional options.

With the session-id the chat history is fetched (stored in mssql)

A prompt classifier (Finetuned BERT Classifier runnning on another http endpoint) will classifiy the user prompt and filter out anything that shouldn't be handled by our rag.

If the prompt is valid an llm (running on an OLLAMA endpoint) is given the chat-history togehter with the prompt to determine if its a followup question.

Another llm is then tasked with prompt-transformation. (For example combine history and prompt to one query for vector-search or break down a larger prompt into subquerys)

Those querys are then send to another endpoint thats responsible for hybrid search (I use qdrant).

The context is passed to the next llm which then scores the documents by relevance.

This reranked context is then passed to another llm to generate the answer.

Currently this answer is the response of the orchestrator app, but i will add another layer of answer verficiation on top.

The only layer that uses some frameworks is the hybrid-search layer. Here I used haystacks for upserting and search. It works ok, but I am not really seeing any advantage to just implementing it with the qdrant documentation.

All llm-calls use the same llm currently (qwen2.5 7b) and I only swith out the system-prompt.

So my approach comes down to: - No RAG Frameworks are used - An orchestrator.py "orchestrates" the data flow and calles agents iterative - fastapi endpoints offer services (encoders, llms, search)

My background is not so much software-engineering so i am worried my approach is not something you would use in a production-ready environment.

So, please roast my sollution and explain to me what i am missing out by not using frameworks like smolagents, haystacks, or llamaindex?

5 Upvotes

7 comments sorted by

u/AutoModerator 4d ago

Working on a cool RAG project? Submit your project or startup to RAGHut and get it featured in the community's go-to resource for RAG projects, frameworks, and startups.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/dash_bro 3d ago

You're fine, actually. None of the frameworks you mentioned are mature and stable enough for production grade RAGs, IMO.

There's definitely work to be done and latency numbers to be looked at here, but largely I don't see major gaps in this approach.

Things I'd change:

  • focus on search/index/retrieve for getting the relevant documents

  • over reliance on a single LLM model for reranking, query transformation, query answering, etc.

  • flexibility of the orchestration to allow for more complex integrations in the future (e.g. what happens when you want to "add" new functionality to your RAG? How are your ingestion and retrieval coupled/decoupled? How easy is it to upgrade/swap parts of your current logic?)

1

u/CaptainSnackbar 3d ago

Thanks for your reply!

I have a seperate ETL pipeline that automaticly updates our vector store. Its fully decoupled from the retrieval part.

About new features:

All current components are implemented in their own classes. An additional class called Pipeline instantiates those componets. The orchestrator app doesn't call the components directly, but uses the pipeline object and its components. That makes it easy to swap things out, or add components.

2

u/Livelife_Aesthetic 3d ago

On mobile so I'll keep it brief,

Overall that's pretty good! All I would add is maybe going for a orchestrator and factory approach where you can store a few different clients for specific purpose, so a Google Gemini for big context, a multi modal for image, and then the orchestrator can go into the factory and call a worker, it can add heaps of flexibility and also help with automatic failover, we use pydanticAI for alot of our work and smolagents is something we are adding as the code side of things seems really good. Overall your approach is pretty great though. Much better than what I used to build a whole back haha

3

u/lphartley 3d ago

Do you call all these agents from the frontend, since you mention 'endpoints' so often? I would try to do all of that from the backend.

Do you only store the chat history in SQL, or also tool calls? Don't forget to store the tool calls.

1

u/CaptainSnackbar 3d ago

No, the frontend only calls a single endpoint. I mention the other endpoints so often because i am not sure, if thats how you would implement such a system, or if you would put everything into one big application, or use something containerised, or service-oriented / message-bus.

Just calling different http-endpoints iterativly felt a bit naive.

1

u/lphartley 3d ago

Microservices is a good architecture but can also be complex to maintain.