GenAI interview questions: RAG, agents and evaluation

GenAI interview questions fall into four areas: retrieval-augmented generation (RAG), the choice among prompting, RAG and fine-tuning, agents and tool calling, and evaluation and production concerns. A strong answer explains the mechanism, names the trade-off and says how the result would be measured. The nine questions below are representative of what a GenAI depth round covers; they are written for practice and are not taken from any company's interviews.

What do GenAI interviews test?

GenAI interviews test whether you can build an LLM application that is reliable, and whether you can tell why it fails when it fails. Naming tools earns little. Interviewers listen for three habits: you break a system into stages, you know the typical failure of each stage, and you measure before and after a change.

The depth round is one of several, and the others are described in the GCC AI interview process. The role these questions belong to is set out in the GenAI / LLM engineer role guide.

How should you structure an answer to a GenAI question?

Structure an answer to a GenAI question in three parts: the direct answer in one sentence, the mechanism or trade-off behind it, and how you would verify it. The third part is the one most candidates leave out, and it is the one that signals production experience.

RAG interview questions

RAG questions are the most common GenAI interview questions, because most enterprise LLM applications answer questions over private documents. Revise the pipeline in how RAG works before practising these.

1. Walk me through how a RAG system answers a question.

• Indexing, done ahead of time: documents are parsed, split into chunks, converted to embeddings, and stored in a vector index with metadata.
• Retrieval, at query time: the question is embedded with the same model, and the nearest chunks are fetched, often combined with keyword search.
• Optional reranking: a more accurate model reorders the candidates so the best few reach the prompt.
• Generation: the model receives the question and the retrieved chunks, with an instruction to answer only from them and to cite them.
• Close with the weak points: parsing quality, chunk boundaries and retrieval misses cause more failures than the generating model.

2. How do you choose a chunk size, and why does chunking matter?

• Chunking matters because retrieval returns chunks, not documents: a chunk that is too large dilutes the embedding with several topics, and one that is too small loses the context needed to answer.
• There is no universal size. Start from the document's structure, such as sections and paragraphs, instead of a fixed character count.
• Use overlap or attach the section heading to each chunk so meaning is not cut at boundaries.
• Decide by experiment: build a small test set of questions with known source passages and compare retrieval hit rate across chunking strategies.

3. Your RAG system returns confident but wrong answers. How do you debug it?

• Gather real failing questions and label the expected answer and source passage.
• Check retrieval first: was the correct passage in the retrieved set? If not, the problem is indexing, chunking, the embedding model or the query.
• If the passage was retrieved but the answer is wrong, the problem is generation: the prompt, too many distracting chunks, or the model ignoring context.
• Typical retrieval fixes: hybrid search, a reranker, query rewriting, metadata filters. Typical generation fixes: tighter instructions, fewer and better chunks, required citations, and permission to say "not found".
• Re-run the whole test set after each change.

4. When would you combine keyword search with vector search?

• Vector search matches meaning, so it handles paraphrases well but can miss exact tokens such as product codes, error numbers, names and abbreviations.
• Keyword search matches exact terms but misses synonyms.
• Hybrid search runs both and merges the ranked lists, which covers both failure modes.
• It is most useful in technical, legal and financial content with many identifiers.

Prompting, fine-tuning and model behaviour questions

5. When would you fine-tune a model instead of using RAG?

• RAG closes a knowledge gap: the model lacks facts that are private or change often. RAG keeps facts updatable and lets answers cite sources.
• Fine-tuning closes a behaviour gap: the model knows enough but must follow a consistent style, format or narrow task, or a smaller model must match a larger one at lower cost and latency.
• Fine-tuning is a poor way to add facts that change, because every change needs retraining.
• The sensible order is prompt engineering first, then RAG if knowledge is missing, then fine-tuning if behaviour is still inconsistent.

6. What causes hallucination, and how do you reduce it?

• A language model generates likely text; it has no built-in check that a statement is true, so it can produce fluent, unsupported claims when it lacks the information.
• Ground the model: supply retrieved sources and instruct the model to answer only from them.
• Allow abstention: an explicit "the documents do not cover this" path removes pressure to invent.
• Require citations and verify them in code where possible.
• Constrain outputs with structured formats and validation, and measure groundedness on a test set. Say plainly that hallucination can be reduced but not eliminated.

AI agent interview questions

7. What is an AI agent, and how does tool calling work?

• An agent is a model running in a loop: it decides on an action, the application executes the action, and the result is fed back until the task is done.
• With tool calling, the application describes each tool by name, purpose and parameter schema. The model returns a structured request to call a tool; the application, not the model, runs it.
• A single model call with retrieval is not an agent; the loop and the model's choice of next step are what distinguish one.
• Agents suit tasks whose steps cannot be fixed in advance. A fixed workflow is cheaper, faster and easier to test when the steps are known.
• More detail is in how AI agents work.

8. What goes wrong with agents in production, and how do you guard against it?

• Loops and runaway cost: cap the number of steps and the spend per task.
• Wrong tool or bad arguments: validate every tool input against its schema and return clear errors the model can act on.
• Compounding errors: a small early mistake grows over many steps, so keep tasks short and add checkpoints.
• Prompt injection: text in a retrieved page or tool result may contain instructions. Treat tool output as data, give tools the least privilege they need, and require human approval for irreversible actions.
• Hard debugging: log every step, input and output so a failed run can be replayed.

LLM evaluation and production questions

9. How do you evaluate an LLM application before and after launch?

• Build a test set from real or realistic queries, including difficult and out-of-scope ones, with expected answers or grading criteria.
• Evaluate stages separately. For RAG: retrieval metrics such as whether the right passage appears in the top results, then answer metrics such as correctness and groundedness.
• Use code-based checks where the output is structured, human review for a sample, and a model as a judge for scale, after checking that the judge agrees with human graders.
• Run the test set on every prompt, model or index change as a regression test.
• After launch, monitor user feedback, sampled conversations, cost and latency, and feed new failures back into the test set.
• The methods are explained in LLM evaluation basics.

How do you prepare for GenAI interview questions?

Prepare for GenAI interview questions by building one small application and measuring it, then practising answers aloud. Reading answer outlines gives you vocabulary; a project gives you the follow-up answers that outlines cannot.

GenAI roles still include general machine learning questions, so pair this list with machine learning interview questions, and expect a design problem as covered in the ML system design interview framework. To check your GenAI skills against what the role expects, take the free readiness check on The GCC Engineer.