What is RAG architecture?
A RAG system has 2 connected paths:
The offline ingestion and indexing path, which prepares documents for retrieval.
The online query and generation path, which retrieves information and produces an answer.
The distinction matters. A document can be correctly indexed and still fail to support a user query if the retrieval logic, metadata filters, or context selection are poorly designed.
How RAG differs from a standalone LLM
A standalone large language model generates an answer from its training and the instructions provided in the request. A RAG system retrieves external information at query time and supplies it as context.
This gives organizations a way to work with internal documentation, product information, policies, technical manuals, research, or support content. The source material can be updated through the ingestion process without retraining the language model for every document change.
RAG can improve grounding and traceability, but it does not guarantee factual answers. The result still depends on source quality, retrieval accuracy, context selection, and the model’s ability to use the supplied evidence.
Organizations planning a domain-specific AI system can use machine learning engineering to connect retrieval architecture with business requirements, data sources, and production constraints.
How the RAG pipeline prepares knowledge
The RAG pipeline begins before a user asks a question. It transforms source content into retrieval-ready units.
1. Document ingestion and parsing
The system first connects to sources such as file repositories, websites, databases, knowledge platforms, or business applications.
Parsing converts these sources into searchable content. It should preserve useful structure, including:
Headings
Sections
Tables
Page numbers
URLs
Document titles
Publication dates
Ownership information
Access permissions
PDFs, scanned documents, images, and complex tables may require additional extraction or optical character recognition. Poor parsing can damage the source before the retrieval process even begins.
2. Chunking and metadata
Large documents are divided into smaller retrieval units called chunks. Each chunk should contain enough local context to be meaningful while remaining small enough to embed and include in a model request.
There is no universal chunk size. A policy document may work well when split by section, while a technical manual may require a structure based on headings, procedures, or page boundaries.
Metadata gives each chunk additional context. It can include the document source, author, date, department, language, product, region, or security classification.
3. Embeddings and indexing
Embeddings are numerical representations of content. They allow the system to compare the meaning of a user query with the meaning of stored document chunks.
The resulting vectors are stored in a vector database or another vector index, alongside the original text and metadata. The index can then return content that is conceptually related to the query, even when the wording is different.
When source documents change, the system may need to reprocess affected chunks and update their embeddings. Freshness therefore depends on the source synchronization process, the indexing process, and the time required for new data to become searchable.
Vector database, embeddings, and semantic search
The retrieval layer connects the user’s question with the knowledge base.
What a vector database contributes
A vector database stores embeddings and supports similarity searches. It may also store the original text, metadata, document identifiers, and access-control information.
The vector database is one component of the RAG architecture. It does not replace document processing, metadata management, security filtering, prompt assembly, or response evaluation.
The choice of index structure affects retrieval speed, storage, filtering, and update behavior. The architecture should follow the size of the knowledge base, query volume, access rules, freshness requirements, and expected response time.
How semantic search works
Semantic search retrieves information according to meaning rather than exact word matches. A user asking about “employee leave rules” may retrieve a document titled “annual vacation policy,” even when the query and source use different terms.
Semantic search works well for natural language questions and domain content with varied terminology. Exact keyword search still matters for product identifiers, error codes, legal references, part numbers, and version names.
When hybrid retrieval and reranking help
Many RAG systems combine vector search with keyword search. This approach, known as hybrid retrieval, helps the system handle both conceptual queries and exact terms.
A reranker can then review the first group of retrieved results and reorder them according to their relevance to the query. Reranking improves the order of available candidates. It cannot recover information that the initial retrieval step failed to find.
Document retrieval and the context window
Once the system receives a user query, it begins the online path.
From the query to the retrieved passages
The system may first clean or rewrite the query, apply filters, and create a query embedding. It then searches the vector index, keyword index, or both.
The retrieval step usually returns a limited number of documents or chunks. This is often called top-k retrieval. The results may then be filtered, reranked, deduplicated, or grouped by source.
Document retrieval should consider more than similarity score. The system may also need to check:
Document freshness
User permissions
Source authority
Geographic or business scope
Language
Version
Content type
A passage can be highly similar to the query and still be outdated, incomplete, or outside the user’s access rights.
Selecting context for the model
The selected passages are placed inside the model’s context window, together with the user request, system instructions, conversation history, and source metadata.
The context window has a finite token capacity. Sending more documents does not automatically improve the answer. Irrelevant, duplicated, or contradictory passages can increase latency and make it harder for the model to identify the evidence that matters.
Good context selection balances coverage with relevance. The system should retrieve enough information to answer the question and remove material that adds noise.
Prompt assembly and grounded generation
Prompt assembly combines the instructions, user question, retrieved evidence, and source information into a structured model request.
The response can include citations or links to the source documents. It should also have a defined behavior for missing evidence, such as stating that the available documents do not support a reliable answer.
A grounded answer should reflect the retrieved context. The system should not describe an answer as fully reliable only because it contains a citation. The citation must support the associated claim.
Evaluating a RAG architecture in practice
A RAG system needs separate evaluation for retrieval and generation.
Measuring retrieval quality
Retrieval evaluation asks whether the system found the right content. Useful checks include:
Relevance of retrieved passages
Coverage of the required information
Recall at a selected top-k value
Duplicate or contradictory results
Freshness of the retrieved content
Permission filtering
Retrieval latency
A response can be fluent while the retrieval layer returns weak evidence. Evaluating the 2 stages separately makes this problem easier to identify.
Measuring generated responses
Generation evaluation asks whether the answer used the retrieved context accurately and completely.
Teams can review:
Factual correctness
Groundedness
Completeness
Citation accuracy
Unsupported claims
User satisfaction
Task completion
Response time
Inference cost
A test set should include realistic user questions, ambiguous requests, questions with no answer in the knowledge base, and queries involving restricted documents.
Common failure points
RAG systems often fail because of:
Poor document extraction
Inconsistent chunking
Weak metadata
Outdated indexes
Incorrect permissions
Low-quality embeddings
Irrelevant retrieval
Excessive context
Missing citations
Conflicting source documents
These issues require different responses. Poor retrieval may require changes to chunking, metadata, filters, or search methods. Unsupported answers may require stronger instructions, better source selection, or an explicit abstention rule.
Organizations can use machine learning engineering when they need support defining evaluation criteria, reviewing retrieval quality, or preparing a RAG prototype for production use.
Security, freshness, and operational control
Enterprise RAG systems handle information that may have different owners, confidentiality levels, and access rules.
Permission checks should happen during retrieval. The system should filter documents according to the user’s identity, group, tenant, role, or document-level access before the content reaches the model. Prompt instructions cannot replace authorization controls.
Freshness also needs monitoring. A source repository may contain the latest policy while the search index still contains an older version. Teams should track ingestion failures, synchronization delays, indexing lag, and deleted or modified documents.
The architecture should also keep provenance. Each retrieved chunk should be traceable to its source document and location. This supports review, debugging, compliance, and user trust.
When machine learning engineering supports RAG projects
RAG projects become more complex when information is spread across several systems, access rules vary by user, documents change frequently, or answer quality is difficult to measure.
External support can help with:
Knowledge-source assessment
Ingestion and indexing design
Embedding and retrieval strategy
Vector and hybrid search architecture
Permission-aware document retrieval
Evaluation datasets and quality reviews
Citation and provenance design
Integration with business applications
Production-readiness planning
A RAG system is more than a language model connected to a vector database. Its quality depends on the entire path from document ingestion to generated response. Parsing, chunking, embeddings, semantic search, document retrieval, context selection, access control, and evaluation all shape what users receive.
Mantu’s machine learning engineering can support organizations that need to connect AI architecture with business applications, data sources, and production requirements.
