AI engineer interview questions to ask before you hire

The most useful AI engineer interview questions ask a candidate to explain decisions they made on a real system: why they chose a model, how they measured quality, what it cost to run and how they kept data safe. Good answers contain numbers, failures and trade-offs. Weak answers contain tool names.

These questions are for a business owner or tech lead hiring an AI developer, as an employee or through an agency. You do not need to be an engineer to use them. You need to listen for specifics.

How should you run the interview?

Start with one past project. Ask the candidate to walk through a system they built that real users relied on, then use the questions below to probe it. Someone who did the work can go three levels deep on any decision; someone who watched it being done runs out after one.

Then give them a small version of your own problem: twenty anonymised examples of the task, such as customer emails to sort or invoices to read, and ask how they would approach it. If you are hiring an agency rather than a person, put these questions to the engineer who will write the code, as our checklist for hiring an AI development company suggests.

Questions on LLMs and RAG

1. "When would you use retrieval (RAG), fine-tuning, or a better prompt?" A good answer starts with the cheapest option: a clear prompt with a few examples, then retrieval when the model needs your documents, and fine-tuning only for a fixed output format or vocabulary that prompts cannot fix.
2. "How did you split and index the documents, and how did you know it worked?" Look for a splitting choice tied to the document type (a contract clause, a product page, an FAQ answer) and a check that the right passage was retrieved, measured separately from whether the final answer was right.
3. "What does the system do when the answer is not in the documents?" It should say so, and the candidate should have tested that case on purpose.
4. "Which model did you choose, and what did you compare it with?" A good answer compares at least two, often a hosted model and an open-weight one, on the same test cases, and weighs speed and running cost alongside quality.

Questions on data and evaluation

This group separates engineers who ship from engineers who demo.

5. "How did you measure quality before launch?" You want to hear about a fixed set of real cases with known correct answers, scored again after every change. Anthropic's evaluation guide asks for tests that "mirror your real-world task distribution", edge cases included. Our glossary entry on model evaluation suggests 50 to 200 cases as a practical start for a narrow task.
6. "How did you score free text answers?" Exact match works for fields and labels. For open text, a good answer mentions a person reviewing samples, a checklist, or a second model acting as judge, with its scores spot checked by a person because AI judges have biases.
7. "Tell me about a time the scores got worse after a change." Everyone who has run evaluations has a story like this. A good one explains how they found the cause.
8. "What data would you need from us, and can you start without personal data?" A good answer asks for samples first, proposes masking or synthetic examples, and asks who owns the data.

Questions on deployment and cost

9. "What did one request cost, and what was the monthly bill?" Someone who ran a system in production knows its cost per request or per document and what drove it: tokens, retries, long prompts, model size. A vague answer here usually means the system never left the demo.
10. "How did you keep response times acceptable?" Listen for streaming, caching repeated prompts, a smaller model for simple steps, or batching work that can wait.
11. "What happens when the provider updates or retires the model?" Good answers mention pinning model versions, re-running the test set before switching, and a fallback model.
12. "How do you know it still works three months later?" Logging, a sample of live outputs reviewed on a schedule, and cost tracked per feature so a runaway bill is caught early.

Questions on security and privacy

The OWASP Top 10 for LLM Applications (2025 edition) is a useful map for this part. Its list opens with prompt injection and sensitive information disclosure, and it also covers excessive agency and unbounded consumption. A strong candidate will know it, or will describe the same risks in their own words.

13. "How would someone trick this system, and what stops them?" A good answer treats every document, email or web page the model reads as untrusted input, keeps instructions apart from data, and does not rely on the prompt alone for protection.
14. "What can the AI do without a person approving it?" Look for a short list of allowed actions, narrow permissions for each tool, and a person approving anything that moves money, deletes records or contacts a customer.
15. "Where does personal data go, and who can see it?" A good candidate asks which law applies to you (India's DPDP Act 2023, UK GDPR or another), where data is processed, whether the provider's terms keep it out of model training, and how long logs are kept.
16. "How do you stop one bug from running up a large bill?" Rate limits, spending caps per key and alerts.

Red flags in the answers

• Tool names with no decisions: "we used a vector database and an agent framework", with no reason why.
• An accuracy figure with no test set behind it.
• No idea what the system cost to run each month.
• "The model takes care of security."
• No story of the system ever failing.

None of these rules a candidate out on their own, but two or three together usually mean the experience is thinner than the CV.

Before you hire, define the first job

These questions work best when you know what the first piece of work is, because you can ask each candidate how they would build that. If you are not sure which process to start with, our AI strategy consulting ends with a written first pilot spec covering scope, success measure, data needed and who reviews the output. You can hand it to us or to anyone else, including a candidate as a take-home brief. If you would rather have a team build it, see how we run custom AI systems projects.

Book a consultation.