What is RAG? Retrieval-augmented generation explained

RAG, short for retrieval-augmented generation, is a method for building LLM applications in which the system looks up relevant passages from your own documents and hands them to the model along with the question. The model then writes its answer from that supplied context instead of relying on what it memorised in training. If you have asked what is RAG because it appears in every GenAI job description, the reason is simple: it is the standard way to make a general model useful on private, changing company knowledge.

RAG sits in the middle of the AI engineer roadmap. It depends on embeddings, vector search and prompting, and it is the prerequisite for agents.

Why do LLM applications need RAG?

LLM applications need RAG because a model on its own knows only what was in its training data. It has not read your policy documents, your product catalogue or yesterday's ticket. Asked about them, a model either says it does not know or produces a plausible invention.

RAG solves three problems at once. Answers can use private information without retraining anything. Knowledge stays current, because updating the document index updates what the system knows. And answers can point to the passages they came from, so a user or auditor can check them. That last property matters a great deal in regulated settings such as banking and healthcare.

How does a RAG pipeline work?

A RAG pipeline works in two phases. Indexing happens ahead of time: documents are split, converted to embeddings and stored. Answering happens per question: the system retrieves the best passages, builds a prompt and generates a reply.

An embedding is a list of numbers that represents the meaning of a piece of text, arranged so that texts with similar meaning have vectors that point in similar directions. Vector search finds the chunks whose embeddings are closest to the question's embedding.

How should you chunk documents for RAG?

Chunk documents so that each chunk is a self-contained unit of meaning, following the document's own structure wherever possible. There is no universally correct chunk size; it is a trade-off you tune against your own questions.

Small chunks retrieve precisely but may lack the context needed to interpret them: a sentence saying "the limit is doubled in this case" is useless without the preceding paragraph. Large chunks keep context but dilute the embedding, since one vector now has to represent several topics, and they use up more of the prompt.

Practical rules that hold up:

• Split on headings, sections and paragraphs before falling back to a fixed length.
• Overlap neighbouring chunks slightly so that a fact on a boundary is not cut in half.
• Attach the document title and section heading to each chunk, so the chunk makes sense alone.
• Treat tables, code and lists as units; splitting a table by character count destroys it.
• Keep metadata with every chunk. Filtering by date, product or permission is often worth more than a better embedding.

What are retrieval and re-ranking?

Retrieval is the step that selects candidate passages for a question, and re-ranking is a second, more careful pass that reorders those candidates. Retrieval must be fast across the whole collection; re-ranking can afford to be slower because it looks at only a short list.

Two retrieval methods complement each other. Vector search matches meaning, so "how do I get my money back" finds a passage about refunds. Keyword search matches exact terms, which vector search handles poorly: product codes, error numbers, names and rare abbreviations. Hybrid search runs both and merges the results.

A re-ranker reads the question and each candidate passage together and scores how well the passage answers the question. That is more accurate than comparing two separately computed vectors, and it lets you retrieve a wide net of candidates and pass only the strongest few to the model.

How do you evaluate a RAG system?

You evaluate a RAG system by measuring retrieval and generation separately, using a fixed set of test questions with known answers. A single overall score cannot tell you whether a wrong answer came from missing passages or from a model that misused good ones.

For retrieval, the question is whether the passages that contain the answer were returned. Recall at k is the usual starting metric:

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Share of the relevant chunk ids that appear in the top k results."""
    if not relevant:
        return 0.0
    hits = len(set(retrieved[:k]) & relevant)
    return hits / len(relevant)

For generation, two questions matter. Faithfulness: is every claim in the answer supported by the retrieved passages? Answer relevance: does the reply address what was asked? Both usually need a human or a carefully checked model as judge; how to evaluate an LLM application explains how to build the test set and where automated judges go wrong.

What are the common RAG failure modes?

The common RAG failure modes fall into retrieval failures, generation failures and data failures, and each has a different fix. Diagnosing which one you have is the core practical skill.

One more risk deserves a mention. Retrieved text is untrusted input: a document can contain instructions aimed at the model. Systems that act on retrieved content need the guardrails discussed in AI agents and tool use.

When should you use RAG instead of fine-tuning?

Use RAG when the problem is missing knowledge, and fine-tuning when the problem is behaviour. The two address different gaps and are often combined.

A sensible order is prompting first, then RAG, then fine-tuning only if a measured gap remains. Each step costs more to build and maintain than the one before.

What do interviews ask about RAG?

Interviews ask about RAG as a system to be designed and debugged, not as a definition. Expect to be given a scenario, such as a support assistant over a large policy library, and asked how you would chunk the documents, which retrieval method you would use and why, how you would know it works, and what you would do when a specific answer is wrong.

Strong answers move through the pipeline in order and name a trade-off at each step. The typical questions and how to structure answers are collected in GenAI interview questions on RAG, agents and evaluation. RAG is a core skill for the GenAI and LLM engineer role and for AI backend engineers, and a small RAG system with a real evaluation is one of the most useful AI portfolio projects you can build.

To see where RAG falls among the skills your target role needs, take the free readiness check and get a personal roadmap in order.