These are the questions AI engineering interviews keep returning to, grouped by topic. Each answer shows what a strong response covers — not a script to memorise. The best answers point to something you built: practise answering each one using your own capstone and lab evidence.
Interviewers are usually testing three things: whether you understand how the system works, whether you can find where it fails, and whether you make tradeoffs with evidence. Use the linked lessons to revisit any answer you cannot give confidently.
Fundamentals
1. What does an AI engineer do that a software engineer or ML engineer does not? An AI engineer builds and operates applications around existing models: shaping the task, connecting data, validating outputs, measuring quality, and running it in production. An ML engineer focuses more on training and serving models; a data scientist on analysis and experiments. Real roles overlap, so describe responsibilities rather than titles. → Roles compared
2. What is a token, and why does it matter to an application? Tokens are the model-specific units text is split into. Cost, latency, and the context limit are all counted in tokens, and the count depends on the model and language — so character limits are only a product choice, not a token budget. Strong answers mention reserving space for the response. → Tokens and inference constraints
3. What does temperature change? Does temperature 0 make answers correct? Temperature reshapes the sampling distribution: lower values reduce variation. It does not make an answer true, and it does not guarantee identical output across systems or versions. Correctness comes from evidence and validation, not decoding settings.
4. Explain embeddings and cosine similarity in plain words. An embedding turns text into a vector so that related meanings point in similar directions. Cosine similarity compares direction, ignoring length. A high score means "related", not "correct" or "allowed to read". → Vectors and similarity
Building with LLMs
5. How do you get reliable structured output from a model? Define the smallest output contract, request it using the provider's structured-output feature, and still validate every field in your code: allowed values, types, lengths, and business rules. Schema-valid output can still be wrong, so measure accuracy separately from format validity. → Structured outputs
6. The model returns invalid JSON 3% of the time. What do you do? Never let invalid output trigger an action. Return a controlled fallback, allow at most one bounded repair attempt that passes the same validator, log the failure category, and track the rate across prompt and model versions. Do not extract a JSON-looking substring and trust it. → Prompt contracts
7. When would you use prompting, RAG, or fine-tuning? Missing or changing knowledge → retrieval. Unclear task → better instructions and examples. Repeated style or format failures that survive a measured prompt baseline → consider adaptation. Fine-tuning does not keep private facts current. Strong answers start from a failure taxonomy, not a preferred technique. → When fine-tuning earns its complexity
Retrieval and RAG
8. Walk me through a RAG pipeline. Ingest documents with stable IDs, versions, and access scope; split them at meaningful boundaries; index them; retrieve authorized candidates for a question; generate an answer citing only retrieved IDs; validate citations and support; return an explicit "unsupported" state when evidence is missing. → Retrieval and grounded answers
9. Your RAG system gives wrong answers. How do you debug it? Separate the two failure points. First check retrieval: was the right passage in the top results? Then check generation: did the answer stay faithful to the passages it had? Inspect case-level traces instead of tuning the prompt blindly. Many "hallucinations" are retrieval or stale-data problems.
10. Why use hybrid search? Lexical search finds exact codes, product names, and identifiers; dense search finds paraphrases. Combine candidates with rank fusion rather than adding incomparable raw scores, then rerank a small set. A reranker cannot recover a document the candidate stage never found. → Hybrid retrieval
11. A policy document was updated. How do you stop the old version being used? Version every document and chunk, build the new version in staging, switch the active version atomically, and remove or filter superseded chunks — including from answer caches. Deleting the source file alone leaves old chunks searchable. → Document pipelines
Evaluation
12. How do you know a prompt change made things better? Run the old and new versions on the same frozen cases, with expectations written before tuning. Report improved, regressed, and unchanged cases — not just an average — and keep a held-out set you did not tune on. → Evaluation and system design
13. Can you use an LLM as a judge? Yes, with care. Calibrate it against human-labelled cases, check for position and verbosity bias, and keep deterministic checks for things like valid citations and forbidden actions. A judge's explanation is not ground truth. → Rubrics and release gates
14. Version B scores higher overall but fails one permission test. Ship it? No. Security and access boundaries are hard gates; an average cannot compensate for a data leak. Block the release, fix the boundary, and add the case to the regression suite.
Security
15. What is prompt injection, and how do you defend against it? Untrusted content — a user message, a retrieved document, a tool result — tries to change the system's behaviour. Labelling text helps the model, but real defence is in the application: authorize before retrieval, give the model only narrow tools, never put secrets in context, and validate outputs before any action. → Trust boundaries
16. How do you stop one customer seeing another customer's data? Derive tenant and user from verified server identity — never from model output or request JSON — and filter retrieval before ranking and generation. Check every crossing point: caches, conversation history, traces, exports, background jobs. Filtering after generation is too late.
Production
17. How do you reduce cost without hurting quality? Measure cost per accepted result, including retries and failures, then try changes one at a time against your evaluation gates: smaller models for easy routes, correctly scoped caching, shorter context, batching where latency allows. Accept an optimisation only if quality gates still pass. → Latency and cost
18. The model provider is down. What should users see? A designed fallback: search-only results, a queued request, or the manual workflow — plus a clear message. Timeouts, bounded retries within one overall deadline, and a switch to disable generation should already exist. → Deployment and operations
19. How do you roll back an AI release? Treat a release as a set: application code, model, prompt, retrieval index, and schema versions. Rolling back may mean reverting the prompt or index alias as well as code. Rehearse it before you need it. → Release with a rollback path
System design
20. Design a support assistant that answers from internal documents for many customers. Cover: identity and tenant isolation; versioned ingestion with deletion; hybrid retrieval with authorization before ranking; a validated, cited answer contract with an "unsupported" state; an evaluation set with hard gates; deadlines, fallbacks, and cost tracking; and a release and rollback plan. Start with the simplest version and say what evidence would justify each extra component. → Build a knowledge assistant end to end
How to practise
Answer each question out loud in under two minutes, then name one artifact from your portfolio that proves it. Ask a peer to follow up with "what if that fails?" — that is where interviews are decided.
For more practice, try the interview practice mode. If you are an Android engineer moving into AI, the Android AI Interview Handbook covers 150 more questions on on-device and cloud AI. To make sure your resume shows these projects well, book a Resume Review.