How to Actually Identify AI Talent in an Interview

Most AI interviews test tool knowledge. The best AI practitioners are not defined by which tools they know but by what they do when the tool fails, the output is wrong, or the problem does not fit a known pattern.

The AI interview as it is commonly conducted is a weak filter for the capability it claims to measure. A standard AI interview asks candidates to list the tools they have used, describe their experience with large language models, and sometimes walk through a toy problem involving prompt construction. The candidates who perform well in this format are the candidates who have optimized for this format, not necessarily the candidates who are most capable of doing the actual work.

The Stanford HAI AI Index Report 2024 documented that AI-related job postings grew faster than overall tech postings, yet practitioner accounts consistently describe a gap between the volume of AI applications and the number of hires who perform as expected. Part of that gap is a hiring process problem. Organizations are using interview formats designed for other skill types to assess a capability that requires different evaluation methods.

What Not to Ask

Three categories of question dominate AI interviews and should be minimized or eliminated because they produce signal-to-noise ratios that are too low to be useful.

Do not ask about certifications. As discussed in Part 2 of this series, certifications measure studied documentation, not applied judgment. A candidate with four AI certifications and a candidate with none may have the same underlying capability or different capability in either direction. The certification answer tells you almost nothing about the relevant dimension.

Do not ask for a tool list. Which tools a candidate has used is a question that every candidate can answer impressively, because tool access is nearly universal. The length and sophistication of the tool list correlates with how much time the candidate has spent in the AI ecosystem, not with how effectively they have used it.

Do not ask "how long have you been working with AI?" In a domain where the relevant knowledge base turns over rapidly, this question may actively mislead you. Three years of experience that includes an extended period of not actively updating is worth less than one year of intensive, current practice. Experience in calendar years is a noisy proxy for the capability you actually want to evaluate.

The Portfolio Review

The most useful starting point for an AI interview is a portfolio review: a structured conversation about real work the candidate has done, with enough specificity to evaluate the quality of the approach.

Ask the candidate to bring one example, ideally something from the past six months, of a workflow they changed using AI. Not "something you're proud of" or "your best work." A specific, recent workflow change. Ask them to walk you through: what the workflow looked like before, what problem they were trying to solve, what approach they took, what did not work initially, what they changed, and what the outcome was.

You are not evaluating whether the workflow change was impressive in absolute terms. You are evaluating the quality of the narrative. Candidates who have genuinely restructured workflows can tell this story with specific detail: the specific failure modes they encountered, the specific iterations they made in response, the specific things they built that persist and are still in use. Candidates who have used AI occasionally produce vague stories: "I used AI to help with research and it saved a lot of time." The specificity of the narrative is the signal.

The interview question that surfaces the most about a candidate's actual AI capability is: tell me about a time the AI was wrong, what you noticed, and what you did about it.

The Decomposition Test

Give the candidate a vague, real business problem from your domain. Not a clean case study. Something genuinely ambiguous: a problem with multiple interpretations, unclear constraints, and no obvious right answer. Tell them you want to see how they would approach it before they touched any tool.

Watch what happens. Candidates who are effective AI practitioners will do something recognizable: they will ask clarifying questions to narrow the problem space, propose a decomposition of the problem into more tractable subproblems, identify the dimensions they would need to evaluate a good answer against, and flag the assumptions that would change their approach. They will do this before mentioning a tool.

Candidates who are occasional users will propose a tool approach immediately. "I would use Claude to draft a response" or "I would prompt GPT-4 with the problem and iterate." This is not wrong, but it is premature. It reveals that the candidate's mental model starts with the tool rather than the problem. Effective practitioners start with the problem and arrive at the tool after the decomposition is clear.

The decomposition test is particularly useful because it is not gameable through credential optimization. A candidate who has never been asked this kind of question but who genuinely decomposes problems before prompting will pass it naturally. A candidate who has memorized frameworks but does not use them in practice will produce something that looks like a decomposition but lacks the specificity and domain sensitivity of a genuine one.

The Failure Case Interview

Ask the candidate: tell me about a time the AI gave you a wrong answer or a misleading one. What did you notice? How did you know it was wrong? What did you do?

This question has a decisive advantage over most AI interview questions: it cannot be answered well by someone who has not actually encountered AI failures in consequential contexts. The answer requires three things that credential optimization cannot produce: a real failure, the domain knowledge to recognize it as a failure, and a diagnostic response to it.

Effective practitioners have a rich library of failure stories. They remember the specific conditions under which the model failed, the specific dimension where the output was wrong, and the specific insight they developed about when to trust the model and when not to. They often describe the failure story with more energy and specificity than the success stories, because the failures were the moments of genuine learning.

Occasional users struggle with this question. They may produce a generic story about "hallucinations" or "inaccurate responses" without specificity about the domain, the stakes, or what they did in response. Or they describe the failure as something the tool did to them, rather than as information they used to improve their approach. The relationship to failure is itself diagnostic: effective practitioners treat AI failures as data about the system; occasional users treat them as nuisances.

The Iteration Story

Ask the candidate to walk you through a prompt or workflow they built over multiple iterations. What did the first version look like? What did not work? What changed between version one and the version they ended up with? Why did each change happen?

This question reveals the iterative habit described in Part 3 of this series. Candidates who iterate in genuine loops can reconstruct the sequence of changes and the reasoning behind each one. They remember what they were testing for in each iteration, what the output told them about the previous framing, and how that informed the next change. The story has a logical arc: each iteration follows from a specific diagnosis of the previous one.

Candidates who do not iterate in this way produce a different kind of story: they tried a few different prompts, one of them worked better than the others, and they kept that one. There is no diagnostic arc. The iterations are random variation rather than designed experiments. This is not necessarily a disqualifying answer, but it is a meaningful signal about the candidate's ceiling: someone who iterates randomly will plateau faster than someone who iterates with a clear hypothesis about what they are testing.

Red Flags and Green Flags

Structuring the Full Interview Loop

An effective AI interview loop for a senior role typically involves four stages, each targeting a different dimension of the capability you are trying to assess.

The first stage is the portfolio review, conducted asynchronously before any live interview. The candidate submits one recent, specific example of a workflow they changed using AI, with a before/after description and a measurable outcome. This screen is low-cost for the organization and high-signal because it requires the candidate to produce actual evidence. It also filters efficiently: candidates who cannot produce this have already told you something important.

The second stage is the decomposition exercise, conducted live with one or two interviewers. Give the candidate a vague, real business problem and 15 minutes to work through how they would approach it before touching any tool. Listen for whether they ask clarifying questions, identify the dimensions of a good answer, name the most likely failure modes, and arrive at a tool approach as a conclusion rather than a starting point. This exercise is not about finding the right answer. It is about observing the quality of the process.

The third stage is the failure case conversation, also conducted live. Ask the candidate to walk through a time AI produced a wrong or misleading output in their actual work. What were the conditions? How did they detect the failure? What did they diagnose as the root cause? What did they change in their approach? The richness of this story is the signal, not the domain or the specific tool involved.

The fourth stage is the iteration story: walk me through a prompt or workflow you built over multiple iterations. What did the first version look like? What changed between versions? Why did each change happen? This exercise reveals whether the candidate iterates with deliberate hypothesis-testing or random variation. It also surfaces the scaffolding habit: candidates who have built durable scaffolding can describe it specifically, including why it is structured the way it is.

What to Do When You Cannot Evaluate the Domain

A practical challenge in AI hiring is that the interviewers often do not have the domain depth to evaluate the quality of a candidate's AI work in their specific area. If you are hiring an AI practitioner for a legal or clinical or supply chain domain that your technical team does not understand deeply, the decomposition test and failure case conversation are still useful, but you may not be able to evaluate whether the candidate's domain judgments are correct.

The solution is to include a domain expert in the interview loop, even if they are not an AI practitioner themselves. A domain expert can evaluate the quality of the candidate's domain reasoning without being an AI evaluator. The AI evaluator can evaluate the quality of the candidate's AI reasoning without being a domain expert. The combination of these two assessments, if the interview is structured to give each evaluator a defined role, produces a more reliable overall signal than either evaluator could produce alone. This is, incidentally, exactly the combination that makes a strong AI practitioner: domain depth and AI fluency together are more valuable than either alone.

An additional consideration for senior hiring leaders is calibration across the interview panel. If different interviewers are evaluating different things and there is no shared rubric for what constitutes a strong versus a weak answer to the decomposition exercise or the failure case conversation, the panel debrief will produce noise rather than signal. Building a simple shared rubric, reviewing it with all interviewers before the loop begins, and aligning on what a strong answer looks like for each component dramatically improves the signal quality of the aggregate evaluation and reduces the influence of individual interviewer bias on the final decision.