Taking RAG From a Demo to Production

You’ve written a one-file RAG.

Your pipeline loads the document, split it into chunks, generates embeddings, stores them in a vector database, retrieve relevant context, and sends that context to a language model.

The answers it generates looks good.

You can ask questions, get useful responses, and maybe even add a simple streamlit interface with fastapi as backend.

This architecture looks straightforward:

For a demo, this may be enough.

But the simplicity of that diagram hides several assumptions.

It assumes the documents are correct and will stay unchanged. It assumes the chunks does not separate those information that is necessary to stay together. It assumes the closest vectors contain useful evidence. It assumes the useful evidence fits inside the model’s context window. It assumes the model will use that evidence correctly. It assumes every request need retrieval. It assumes every dependency will be available when a request arrives.

And if the answer is wrong or the request takes thirty seconds, it assumes you can figure out why. Those assumptions are often acceptable in a demo. They become engineering problems when people start depending on the system.

Over the past few weeks, I explored many of these problems while building a support assistant around a finance service use case. This article uses that work as a practical reference, but the goal is broader: to show what changes when a simple RAG pipeline grows into a system that needs to manage changing knowledge, make better retrieval decisions, verify its answers, handle failures, and explain what happened inside each request.

This is not a framework specific tutorial. The tools can change. The underlying problems does not.

I’m assuming you’ve already built a basic RAG and understand embeddings, vector databases, nodes and edges in langchain, and the difference between indexing and querying. If you’re still learning those basics, I’d recommend starting there first. This article focuses on what comes after.

Throughout this article, whenever we add a new component, we’ll answer one question:

What problem does this solve?

If a component doesn’t solve a real problem, we probably don’t need it.

We will follow this idea to the final architecture.

1. What if the knowledge changes?

A simple RAG usually treats its knowledge base like a folder of PDFs.
You load the documents, create embeddings, store the vectors, and start answering questions. Indexing is something you do once before running the application.
Real corpus does not stay still.

But documentation is updated. Policies change. Old procedures are replaced. A document may be corrected, deleted, or split into several new documents. Sometimes information must stop appearing in answers immediately.
If the index does not reflect those changes correctly, the RAG system can return outdated information even when retrieval and generation are working exactly as designed.

So the first big change is this:

One-Time Indexing
        ↓
Managed Knowledge Pipeline

Once people depend on your RAG, you can’t just think of it as “a folder of PDFs” anymore. You need to know what’s currently live, what changed yesterday, what version users are querying, and how to update it without breaking anything.

Every document needs an identity

Suppose your document produces fifteen chunks. Those chunks are stored separately in the vector database, but they still belong to the same source document. When the document changes, the system needs to know which chunks came from the old version, which vectors should be replaced, and which version is currently active. That requires more than storing raw text and embeddings.

Every chunk needs to carry enough information for the system to answer questions like:

Which document did this come from?

Which version?

Is it still active?

{
    "document_id": "merchant_settlement_guide",
    "chunk_id": "settlement_guide_v3_chunk_12",
    "document_version": 3,
    "section": "Settlement Schedule",
    "updated_at": "2026-07-18",
    "status": "active"
}

The exact fields depend on the application, but the principle is consistent: Every vector should still know where it came from.

Metadata helps with source attribution, but it also makes updates, debugging, tracing, and access control much easier.

Do not re-embed what has not changed

Rebuilding the entire index whenever one document changes would work for a small demo but it becomes expensive and slow as the corpus grows.

Content hashing provides a simple improvement. When a document arrives, calculate a hash of its contents and compare it with the hash stored from the previous indexing run. If the content is unchanged, the pipeline can skip it.

For larger documents, the same idea can be applied at the chunk level. Only changed chunks need new embeddings. This helps optimization and reduces unnecessary work and makes indexing more predictable.

Now the indexing pipeline begins to look more like this:

At this point, indexing is no longer a setup script. It is a whole dedicated system responsible for keeping the application’s knowledge current.

Do not expose a partially rebuilt index

Now this is handy if your corpus is very large and it’s changing or updating frequently. Suppose you are rebuilding an index containing thousands of documents, but the job fails halfway through.

If users are reading directly from the index while it is being rebuilt, they may receive a mixture of old and new information. Some documents may be updated while others are not. That can create contradictions that are difficult to detect from the retrieval layer. A safer approach is to build a new version separately:

rag_index_v1  ← currently serving users

Build and validate:

rag_index_v2

Then switch:

rag_index_current → rag_index_v2

The new index is built and evaluated before traffic moves to it. If the update fails, the old version remains available.

This is the same principle used in application deployments and database migrations:

Do not expose a partially completed update unless the system is explicitly designed to support it.

The embedding model also becomes part of the index’s identity.
If your document vectors were created using embedding model A (e.g. google/embeddinggemma-300m), switching only the query side to model B (e.g. BAAI/bge-m3 or Qwen/Qwen3-Embedding-0.6B) may produce invalid or unpredictable retrieval behavior. The query and document vectors may no longer belong to the same representation space.

An embedding-model upgrade is therefore an index migration:

1. Build a new index using the new model.
2. Re-embed the corpus.
3. Evaluate the new index against representative queries.
4. Compare it with the current index.
5. Move traffic only after the new version is trusted.
6. Keep the old version available for rollback.

The knowledge layer is now much more than:

Documents → Embeddings → Vector Database

It needs to answer:

What knowledge is currently active, where did it come from, which version produced it, and how can it be updated without serving inconsistent information?

Before improving the model or adding a verifier, the knowledge itself has to be correct.

2. What if similarity does not mean evidence?

A vector database returns chunks that are close to the query in embedding space.

That is useful, but “closest” does not mean “contains the answer.”
Suppose a merchant asks:

“When will my settlement be credited?”

Your vector search may return chunks about settlement reports, settlement status, settlement delays, settlement schedules, and bank processing.

All of them are semantically related. But only one may directly answer the question. This is the difference between finding related information and finding evidence that can support an answer.

Similarity gets you close. It does not get you right.

Retrieve broadly, then judge more carefully

Our goal here is simple: don’t let weak information reach the model. A common approach is to retrieve a wider candidate set and rerank it:

User Question
      ↓
Vector Search
      ↓
15 Candidate Chunks
      ↓
Reranking
      ↓
5 Stronger Candidates
      ↓
Context Assembly

Vector search is optimized for quickly finding candidates from a large collection.

A reranker performs a more focused comparison between the question and each candidate. It can make a better judgment about which chunks are useful, although it usually adds latency and cost.

Retrieval is no longer one database call. It is an evidence-selection pipeline.

Search by meaning and exact terms

Some questions depend heavily on exact words. A user may ask about a specific transaction ID, error code, bank, product name, policy number, or technical term. Vector search is great at finding similar meaning, but exact keyword matching still matters for things like IDs, error codes, or product names.

That is why many systems use hybrid retrieval:

The vector side helps with semantic similarity. The keyword side helps with exact terms and identifiers.

There is no universal configuration that works equally well for support documentation, legal documents, source code, product catalogs, and research papers. Retrieval should be designed around the knowledge and the questions users actually ask.

Remove impossible results before ranking

The metadata can also improve retrieval. Suppose a user is asking about a specific product version. The retriever may not need to search every document. A filter can restrict the candidate set to active documentation for the relevant product or tenant. This can improve relevance and correctness by preventing outdated, unauthorized, or unrelated documents from entering the candidate set.

The retrieval path becomes:

Question
   ↓
Query Understanding
   ↓
Metadata and Access Filters
   ↓
Vector and/or Keyword Retrieval
   ↓
Candidate Set
   ↓
Reranking
   ↓
Relevance Decision
   ↓
Evidence

The system is now moving from:

Find similar text. to: Find evidence that can answer this question.

What if the knowledge base does not contain the answer?

Your rag still has another problem. A reranker can always return the highest-scoring results, even when every result is irrelevant. If the system asks for five chunks, it may receive five chunks even though the knowledge base contains no useful answer. That is why retrieval often needs a relevance decision. We can add a threshold, a classifier, or another evaluation step to determine whether the evidence is strong enough to answer. The exact threshold should not be copied from another project. Score can vary across models, rerankers, datasets, and retrieval strategies. You should choose it based on the kinds of questions your users ask.

The important question is what happens when the evidence is weak.

The system could:

• say that the available documentation does not contain enough information,
• ask the user to clarify the request,
• search another approved source,
• route the request to a different workflow,
• or escalate it to support.

Returning weak context and hoping the language model handles it is also a decision, but it should be intentional.

3. What should the model actually see?

Let’s say retrieval works well now.

Your system finds useful evidence. It also has system instructions, the user’s question, conversation history, source metadata, and possibly tool results. All of that has to fit inside the model’s context window.

A simple RAG often concatenates everything:

Question
    +
Conversation History
    +
Retrieved Chunks
    ↓
Language Model

That works until the application has more information than the model can accept. The problem is not only avoiding a context-window error. The system also needs to decide:

Which information deserves space?

Retrieval and context assembly are different jobs

Retrieval answers: What information might be relevant?

Context assembly answers: What information should the model actually see?

Those are not the same decision. The retriever may return fifteen useful chunks. The model may have room for only five after accounting for instructions, the user’s message, conversation history, and the expected answer.

This means your application now needs a context budget:

Total Context Budget
        │
        ├── System Instructions
        ├── Current User Question
        ├── Relevant Conversation State
        ├── Retrieved Evidence
        └── Reserved Answer Space

The system should reserve enough capacity for the model to produce an answer. Otherwise, a large input can leave little room for output.

More context is not always better

It is tempting to include every retrieved chunk that fits. But that cloud make the answer worse. Extra context may introduce conflicting information, repeat the same content, distract the model, increase cost, or bury strong evidence among weaker results. The goal is not to maximize the amount of context. The goal is to provide enough high-quality evidence for the model to answer correctly.

So context assembly should:

• remove duplicate or overlapping chunks,
• prioritize stronger reranker scores,
• preserve useful section headings,
• group related evidence,
• select information from multiple sources,
• summarize older conversation history,
• or compress long documents.

The final prompt becomes a controlled artifact:

System Instructions
        +
Current User Question
        +
Relevant Conversation State
        +
Selected Evidence
        +
Source Information
        ↓
Model-Ready Context

The model only sees what the application choose to place inside that context. That makes context assembly part of the system’s behavior and not just a prompt-formatter.

4. Does every request need the full RAG pipeline?

A simple RAG often assumes that every user message follows the same path:

Question
   ↓
Retrieve
   ↓
Generate
   ↓
Return

Real conversations does not work that way.

A user may say: “Hi.” or “Thank you.” or “What can you help me with?”

Those messages usually do not need query embeddings, vector search, reranking, or document-grounded generation. Sending every message through the complete pipeline adds unnecessary latency and cost. It may also produce unnatural responses because the system is trying to retrieve documentation for a greeting.

The application needs a lightweight routing decision:

User Message
      ↓
Request Classification
      ↓
 ┌────┼──────────┐
 ↓    ↓          ↓
Chat  RAG       Tool or
      Search    Special Workflow

The router does not need to be a large agent. It may use simple rules, a small classifier, or a language model, depending on the application.

The important principle is: The system should choose the smallest reliable path that can handle the request.

A greeting may use a direct conversational response. A knowledge question may enter the RAG pipeline. A request requiring live information may use an approved tool. A high-risk request may follow a restricted workflow.

This is one of the first places where a linear pipeline begins to become a workflow.

What if the next step depends on the previous result?

A user may ask an ambiguous question that requires clarification.
The knowledge base may not contain enough information.
The generated answer may fail verification and require another attempt.
A request may need escalation instead of generation.

At that point, the next action depends on what the system has learned so far. A fixed chain becomes difficult to manage because the application now needs to make decisions during execution.

This is where a workflow or graph becomes useful.

The workflow needs shared state

In my merchant support assistant, I used a shared state object to pass information between the planner, retriever, responder, and grader.

The state contained information such as:

Current Query
Plan or Route
Conversation Messages
Retrieved Documents
Selected Source Chunks
Generated Answer
Grader Feedback
Retry Count
Escalation State

Each component reads the state, performs one responsibility, and returns updates.

For example:

• the planner or router interprets the request,
• the retriever adds relevant evidence,
• the responder generates an answer,
• the grader evaluates grounding,
• the workflow decides whether to return, retry, clarify, or escalate.

A simplified graph might look like this:

The graph is important because the request can now follow different paths based on what the system has learned.

A retry is a policy decision

Suppose verification determines that an answer is not sufficiently grounded.

The system could:

1. regenerate the answer using the verifier’s feedback,
2. retrieve a different or wider set of evidence,
3. ask the user to clarify the request,
4. return only the supported portion,
5. say that the documentation is insufficient,
6. or escalate the request.

Those are different policies.

A retry loop is only one possible response.

The important question is:

What should the system do when it cannot produce a trustworthy answer?

The answer depends on the application.

A general assistant may provide partial help. A financial support assistant may need to be more conservative. A high-impact system may need to abstain or escalate rather than continue generating.

The workflow needs to make that policy explicit.

State may need to survive the request

Once the workflow contains conversation history, intermediate decisions, retries, tool results, or long-running steps, state becomes something the system may need to preserve.

In-memory state can be useful during development, but process memory disappears when the application restarts. In a horizontally scaled deployment, different requests may also reach different application instances.

Persistent state allows workflows to be resumed, inspected, and debugged.

This changes the architecture:

The application is no longer one model call. It is a stateful workflow that may be routed, retried, resumed, inspected, or escalated.

5. The answer sounds right. How do you know it is supported?

The model has received useful evidence.

The prompt tells it to answer only from that evidence. It is instructed to not invent procedures, fees, URLs, or unsupported details.

Then it produces a confident, well-written answer.

How do you know the answer is actually supported?

You do not know simply because the prompt told the model to be grounded.

Prompt instructions influence generation. They do not verify the result.

A language model is optimized to produce a plausible continuation. When evidence is weak or incomplete, it may still produce an answer that sounds useful instead of clearly stating that the information is missing.

That is why production systems often separate:

Generation
    ↓
Verification

Different checks answer different questions

We often group several control measures under the word “guardrails,” but they have different responsibilities.

These checks may use different methods and have different failure policies. An input guardrail does not prove that the final answer is grounded. A grounding grader does not necessarily determine whether the answer is safe. A relevance score does not prove that the answer is relevant. Separating these responsibilities makes the system easier to reason about.

Verification creates a second decision point

A simplified flow looks like this:

Retrieved Evidence
        +
User Question
        ↓
    Generate Answer
        ↓
    Verify Answer
       ↙       ↘
   Supported   Unsupported
       ↓           ↓
    Return     Decide What Next

The verifier receives the original question, the retrieved evidence, the generated answer, and a clear evaluation rubric (rules).

It might check whether the answer:

• makes claims not present in the evidence,
• contradicts the evidence,
• leaves out important qualifications,
• or fails to address the question.

A smaller model may be sufficient for this narrower task, but a verifier is still a model-based component with its own errors.

Verification reduces risk. But it does not create 100% certainty.

Failure should lead to an explicit outcome

If the answer fails verification, the system may retry using feedback such as:

“The answer mentions a settlement timeline that is not supported by the retrieved context. Remove the unsupported claim or state that the documentation does not specify the timeline.”

The responder then gets another opportunity to produce a grounded answer. Retries should be bounded. An unbounded retry loop can increase latency and cost without improving the result. After the retry limit is reached, the system should not automatically treat the answer as verified.

It should make an explicit choice:

Verification Failed
        ↓
Retry Limit Reached
        ↓
 ┌──────┼─────────┐
 ↓      ↓         ↓
Abstain Partial  Escalate
        Answer

“Maximum retries reached” is not the same as “answer verified/correct.”

The system should be allowed to abstain

You should not force your rag to answer every question. Sometimes the correct response is: “I do not have enough verified information to answer this. Here is the next support path.”

Sometimes it’s better to not answer, sometimes it’s better to escalate it. A complete escalation path may look like:

Request
   ↓
Risk or Escalation Detection
   ↓
Create Escalation Record
   ↓
Route to Support Queue
   ↓
Attach Conversation and Evidence
   ↓
Notify the User

The system’s purpose is not to generate an answer at all costs. Its purpose is to provide the correct outcome for the request.

6. How does the system survive, explain, and improve?

The RAG workflow is only one part of the application.

It also depends on embedding providers, language-model providers, vector databases, rerankers, application servers, storage, authentication systems, monitoring services, and deployment infrastructure. Each dependency can become slow, unavailable, rate-limited, or overloaded. The application needs to decide what happens when that occurs.

It also needs to answer a second question: “If something goes wrong, can we explain why?”
And a third: “When we change the system, can we prove that it improved?”
These questions connect reliability, observability, and evaluation.

Trace the request from start to finish

Suppose a user reports: “The assistant took thirty seconds to answer.”

Where did those thirty seconds go?
Was query embedding slow?
Did vector search take too long?
Did the reranker process too many candidates?
Was the model provider delayed?
Did verification trigger another generation attempt?

Without tracing, you may only know the total request duration. You know the system was slow. You do not know why.

A complete request can be represented as a trace:

RAG Request
├── Input Validation
├── Request Routing
├── Query Embedding
├── Vector/Keyword Retrieval
├── Reranking
├── Context Assembly
├── LLM Generation
├── Answer Verification
├── Retry or Fallback
└── Response

Each stage can record the information needed to explain its behavior.
For example:

Retrieval
    index version
    filters
    result count
    latency

Reranking
    candidate count
    selected count
    latency

Generation
    model
    context size
    token usage
    latency

Verification
    result
    feedback
    retry count
    latency

In my implementation, major stages were exposed through tracing spans such as:

Query Embedding
Qdrant Vector Search
Semantic Reranking
LLM Synthesis
Hallucination Grader

The names are less important than the visibility they create.

Record the evidence, not only the answer

Now imagine a different report:

“The assistant gave me the wrong answer.”

Was the problem stale information, incorrect chunking, weak retrieval, a reranker that removed the useful chunk, an overloaded context, a model that ignored the evidence, or a verifier that missed the problem?

From the final response alone, you won’t be able find the issue. The identity of the retrieved chunks is often one of the most useful pieces of information. It allows you to differentiate between findings such as:

• The system retrieved chunks from an outdated document version,
• The correct document was retrieved, but the useful section was ranked too low,
• The correct evidence reached the model, but the model generated a claim that was not present in it.

Those findings require different fixes. Without retrieval attribution, every bad answer can look like a hallucination problem, even when the model was faithfully answering from incorrect evidence.

Observability is therefore not only about latency and errors. It is also about understanding why the system believed the answer was relevant.

Users should receive stable, understandable errors. They should not see stack traces, provider details, or internal configuration. Internally, the system should preserve the exception, failed dependency, trace, retry attempts, fallback behavior, and relevant model or index versions.

The user sees a useful response. Operators retain the information needed to investigate.

Evaluate changes instead of trusting intuition

Observability tells you what happened during a request. Evaluation tells you whether the system is improving or getting worse.

Those are related, but they are not the same.

Suppose you change the chunking strategy, embedding model, retrieval parameters, reranker, prompt, or generation model. The application still returns answers. But are the answers better? Without evaluation, it is easy to replace one visible problem with several quieter ones. A useful evaluation set should include the kinds of questions the system actually receives:

• direct factual questions,
• multi-step questions,
• questions requiring multiple sources,
• ambiguous questions,
• questions with no answer in the documentation,
• outdated or superseded procedures,
• short queries,
• long queries,
• and user wording that differs from the documentation.

Not every example needs one perfect reference answer.

Some are better evaluated using properties:

• Was the correct document retrieved?
• Was the answer grounded?
• Did it address the question?
• Did the system abstain when knowledge was missing?
• Did it follow the expected escalation policy?

Retrieval quality and answer quality are different

A good answer requires more than good retrieval.

The system can fail in several places:

Wrong Document Retrieved
        ↓
Retrieval Problem

Correct Document, Wrong Section
        ↓
Chunking or Ranking Problem

Correct Evidence Reached the Model
        ↓
Model Misused It
        ↓
Generation Problem

Correct Answer Generated
        ↓
Verifier Rejected It
        ↓
Verification Problem

These stages should be evaluated separately where possible.

If every metric is reduced to “Was the final answer good?”, it becomes harder to identify which component needs work.

Before deploying a new index, embedding model, reranker, or prompt, a safer process is:

Build New Version
        ↓
Run Evaluation Set
        ↓
Compare With Current Version
        ↓
Review Important Failures
        ↓
Route Limited Traffic
        ↓
Monitor
        ↓
Keep or Roll Back

Manual testing is still useful, but “the new answers look better in a few examples” is not reliable evidence that the system improved.

User feedback closes another loop

Automatic evaluation cannot predict every failure. Users should be report that an answer did not help, that information is outdated, or that they need human support.

Useful signals include:

• thumbs up or down,
• “this did not answer my question,”
• “this information is outdated,”
• repeated questions after an answer,
• escalation requests,
• or support agents correcting a response.

Those responses/feedback become much more valuable when connected to the request trace. A negative rating without the retrieved evidence and execution details is difficult to investigate.

A negative rating that is linked to the trace can show what was retrieved, what the model saw, which index version was active, whether the request was retried, and which model generated the answer.

That turns user feedback into engineering information.

Reliability is a policy, not a collection of retries

Retries can help with temporary failures such as rate limits, timeouts, and brief service outages. But retries are not automatically safe. Retrying immediately can make an overloaded dependency worse. Retrying a request that is unlikely to succeed only increases latency.

A useful policy usually defines:

• timeouts,
• a maximum number of attempts,
• exponential backoff,
• retryable and non-retryable failures,
• and a fallback after the limit is reached.

The important question is not:

How do we add retries?

It is:

Which failures are likely to recover, how long should we wait, and what should happen when recovery fails?

Fallbacks require the same thinking. A backup model may keep the system operating, but it may have lower quality, different context limits, higher latency, or different capabilities.

A fallback is not simply “Primary Model Failed, So Use Another Fallback Model”.

It is a policy decision about which capabilities and quality guarantees can be reduced.

Caching

Imagine 500 users ask the same question today.

Do you really want to generate the same embedding, run the same vector search, rerank the same chunks, and generate the same answer 500 times?

Caching can reduce latency and cost.

Possible cache layers include:

Query Embedding Cache
        ↓
Retrieval Result Cache
        ↓
Final Answer Cache

But caching introduces a new correctness problem. If the corpus changes, an old cached answer may become stale. Cache keys or invalidation rules should therefore account for relevant versions, such as the active index version, document version, model version, user permissions, or conversation state.

Scale the online and offline paths separately

A demo often runs as:

One Application
One Process
Local State
Development Storage

That can be enough during development. But as usage grows, the indexing workload and user-facing workload may need to scale independently.

Background workers can process ingestion and reindexing without competing with user requests. Application instances can then focus on serving users. The exact infrastructure depends on the application, but the principle is simple:
Building the knowledge base and answering questions are different workloads. They should be treated as such.

Security belongs inside the retrieval architecture

A production RAG may expose internal or proprietary knowledge. That makes access control part of retrieval. Users should only be able to retrieve the documents they are authorized to access. This should not be enforced only through prompting. If authorization happens after retrieval, unauthorized information may already have reached the model.

Authorization should be enforced during retrieval itself:

User Identity
      ↓
Permissions and Tenant Scope
      ↓
Authorized Retrieval Filter
      ↓
Only Allowed Evidence
      ↓
Language Model

The model should not receive information the user was never allowed to retrieve.

Security also includes authentication, tenant isolation, secrets management, audit logs, rate limiting, input validation, and protection against prompt-injection attempts.

Retrieved documents should be treated as data, not as trusted system instructions. A document may contain text that looks like an instruction, but that content should not be allowed to override the application’s rules.

These are not optional additions around the RAG system. They are part of how the application behaves in production.

It might be tempting to use GPT-5 or Claude Opus for every step in your pipeline.

But should your reranker really cost the same as your final answer generation?

The model is an important dependency. A production system may use different models for different responsibilities:

Embedding Model
    → Represent documents and queries

Reranker
    → Compare questions with candidate evidence

Generation Model
    → Produce the final response

Verification Model
    → Evaluate grounding and relevance

Classifier
    → Route requests or identify risk

Using the strongest available model for every stage may increase cost and latency without improving the system proportionally. A smaller model may be sufficient for routing, classification, or grading. A stronger model may be reserved for final answer generation.

The right combination depends on the quality, latency, cost, and reliability your application needs.

The part that is not RAG at all

A production RAG is not simply a larger version of the one-file demo.

The original pipeline still exists:

Retrieve → Add Context → Generate

But each part has grown into a system with its own responsibilities.

Static Documents
        ↓
Managed Knowledge and Versioned Indexing

Nearest-Neighbor Search
        ↓
Evidence Retrieval, Filtering, and Reranking

Prompt Concatenation
        ↓
Controlled Context Assembly

One Path for Every Request
        ↓
Routing and Stateful Workflows

Generate and Return
        ↓
Verification, Abstention, and Escalation

Application Logs
        ↓
Tracing, Evaluation, and Feedback

One Process
        ↓
Reliable, Secure, Scalable Infrastructure

Not every application needs every component described here.

A small internal tool may not need multi-provider fallbacks, distributed workers, or complex evaluation infrastructure.
A high-volume support system may need all of them and more.
Production readiness is not a universal checklist where every box must be checked.

It depends on the application:

• How costly is a wrong answer?
• How often does the knowledge change?
• How many users depend on the system?
• How quickly must it respond?
• What information is it allowed to access?
• What should happen when the system is uncertain?
• How much downtime or degraded quality is acceptable?

Those questions determine the engineering work.

The important shift is not adding every possible production component. It is understanding the assumptions inside the simple RAG and deciding which of them are no longer safe.

A one-file RAG can demonstrate that retrieval and generation work together.

A production-grade RAG needs to answer harder questions:

• Is the knowledge current?
• Did we retrieve evidence that can actually answer the question?
• Did the model receive the right context?
• Is the answer supported by that context?
• What happens when a dependency fails?
• If the system is slow or wrong, can we explain why?
• And when the system changes, can we prove that it improved?

Those are the questions that shape the architecture.

The goal is not to build the most complicated RAG system possible. It is to understand where the simple pipeline can fail, decide which failures matter for the application, and build the controls needed to manage them.

Production isn’t a fixed architecture. It’s a series of engineering decisions you take to solve your specific problems and risks your application faces.