How Teams Are Assessing AI Skills in Hiring Today

TL;DR: Companies are trying everything from "prompt engineering trivia" to live vibe coding interviews. The best AI skills assessments look less like quizzes and more like realistic work samples: can the candidate use AI to think, produce, verify, and improve?

Resumes now say the candidate uses AI. Very little in a normal interview loop tells you how well.

The question we get asked most often is how to test that, when every candidate claims it.

At Gradient we talk to a lot of teams about their hiring processes, and there is no standard practice yet. The pattern we see is that the more realistic the assessment, the better the signal, and the more it costs in privacy, fairness, scale, and interviewer time.

Here are the seven approaches we've seen, and what each one tells you.

What to call this

There's no settled term. In the last month we've heard buyers say AI savvy, AI proficient, AI agility, AI native, and "good prompt writers". We say AI fluency. The label matters less than having the definition written down before anyone scores anything.

A scoping note: if you're hiring engineers, HackerRank, CodeSignal, and CoderPad already do this well, and we'd point you there. Sales, ops, marketing, finance, product, and recruiting roles now depend on AI skill too, and much less was built for them.

1. Asking "How do you use AI today?"

This is the most common approach.

Behavioral questions are easy to add to an interview loop, they're consistent across candidates, and require zero setup. A recruiter, hiring manager, or peer interviewer can ask:

• What AI tools do you use?
• How do you use AI in your current work?
• Can you give an example of something AI helped you do faster or better?

But, typically we hear that this is low-signal. Candidates give rehearsed answers, and there's usually not enough time to dig into the specifics. And if the interviewer does not personally use AI deeply, it can be hard to tell the difference between someone who casually uses ChatGPT and someone who has changed how they work.

2. Asking AI or prompt engineering trivia

Another approach is to ask candidates technical questions about AI.

For example:

• What is retrieval-augmented generation?
• What does "LLM as judge" mean?
• How would you evaluate whether an AI-generated answer is correct?
• What makes a prompt effective?

These questions can cut a little deeper than "How do you use AI?" They can reveal whether a candidate understands the vocabulary and basic concepts behind modern AI tools, but they still tend to be shallow.

AI fluency is not the same as knowing terminology. A candidate can define a concept without being especially effective at using AI in real work. The reverse is also true: some excellent AI users may not use the formal language that has developed around LLMs.

Trivia-style questions can be useful as a light screen for technical roles. But for most hiring processes, they shouldn't be treated as a proxy for practical AI skill.

3. Running a live "vibe coding" (AI-assisted work) session

Some companies ask candidates to screenshare while completing a task with AI.

For engineering roles, this often looks like "vibe coding": the candidate uses tools like Claude Code, Cursor, ChatGPT, or Copilot to build or debug something live. For non-engineering roles, it might be a live research, writing, analysis, or operations task.

The big benefit of this method is that you get to see how candidates work in real time: how they frame the problem, what context they give, how they balance speed and quality or verify facts and logic.

The downside is that live AI work sessions can be inequitable and are expensive for team time.

On the equity point, candidates may have different access to paid tools. They may be using personal accounts with private work history, and screen sharing can create privacy concerns, especially if the candidate is using their own laptop and AI accounts.

They also once again require the interviewer to know what good looks like to be able to ask probing questions and process what's happening live. Lastly, this can be expensive, since it requires your most AI-savvy team members to set aside synchronous time for interviews.

4. Asking candidates to show their ChatGPT or AI tool history

Another thing we've started to hear about are companies asking candidates to prompt ChatGPT to describe their working style, strengths, and weaknesses: in other words, letting their AI tool speak directly to how they work.

We strongly discourage this!!

It may seem like a shortcut to understanding how someone uses AI, but it creates serious privacy issues. Personal AI accounts can include sensitive work, personal writing, health questions, financial information, confidential company context, or other material that has nothing to do with the hiring process and could open up legal risks.

It's also limited in other ways: for example, if a candidate switches between Claude and Codex, half of their AI engagement and usage is in another tool. Similarly, if most of someone's AI use is on a work account, they may not be able to participate fully.

To summarize, this is high signal, and a bad idea. AI fluency should be assessed through job-relevant performance, not by inspecting someone's private workspace.

5. Creating a controlled candidate environment

The most interesting tactic we've heard from a company is that an AI-native startup mailed a candidate a laptop that was connected to company internal services so that they could complete a take home, using the company's AI SlackBot to ask questions and pull real corporate context.

This is an interesting approach: everyone gets the same tools, the same task, and the same constraints. It also avoids asking candidates to expose personal accounts. Plus, since the candidate is interacting with the SlackBot, the company can also see what questions they asked as a proxy for thought process.

However, this is very expensive, definitely not scalable, and requires lots of plumbing: shipping boxes, an NDA, and likely to pay the candidate for doing real work.

But, we thought it was interesting and unique, so we've included it here!

6. Running an AI systems or agent design interview

A final option, popular with AI native companies like Sierra, is a live whiteboarding-style interview focused on AI workflows or agent design. In this case, AI isn't actually used, but candidates are asked to show deep systems level knowledge of AI.

This is especially useful for senior candidates, technical operators, product managers, and anyone expected to design AI-enabled workflows rather than simply use AI tools.

The risk is that this becomes too abstract, and the biggest downsides are the drag on team time and on resources for the most AI-native team members, who are best able to lead these types of sessions.

7. Building your own assessment internally

The most common alternative to any of the above is a team building one themselves, and for many teams that is the right move. You control the rubric, you skip procurement, and a first version ships in a sprint.

The longer work arrives later: the rubric and the evals that keep scoring stable, filtering personal information out of candidate sessions, candidate support, staying aligned with hiring law, connecting to your ATS, and re-validating after each model release.

We wrote that up in full, including the cases where building wins: Build or buy an AI skills assessment.

Summary: what an AI fluency assessment should be

To summarize themes, we think the ideal assessments for AI skills should be:

• Fair and equitable: Each candidate should have the same resources
• Privacy-forward: Candidates should not be asked to share private data or their own AI outputs
• Realistic: This should focus on how candidates use AI to do real work, not answer trivia questions.

For companies, it's practically also important to make sure your assessment approach is scalable, doesn't present too much of a drain on team time (especially for your most AI-forward team members) and is high-signal. It's also important to make sure interviewers are well-equipped to identify what good looks like if needed.

What a good AI skills assessment should measure

We believe that a strong AI skills assessment should measure whether a candidate can:

• Frame an ambiguous problem clearly
• Choose the right AI tool for the task
• Give the model useful context
• Break work into effective steps
• Iterate when the first answer is weak
• Verify AI-generated outputs
• Spot hallucinations or flawed reasoning
• Protect confidential information
• Combine AI output with their own judgment
• Produce something useful, not just plausible

This is the difference between AI usage and AI fluency: using AI as a reliable collaborator while keeping taste, judgment, and accountability for the final work.

What the spread looks like in practice

We run this exercise with corporate teams as well as candidates. In one recent session with a large enterprise sales organization, 78% of participants connected at least one data source to their AI tool, 18% used more than one conversation thread, and 3% edited the tool's memory or built a reusable instruction file.

Nearly all of them would have answered yes to "do you use AI at work". An interview question cannot separate the 78% from the 3%, and the gap between them is most of what you are hiring for.

Where Gradient fits

Yes, this is Gradient's corporate blog, so this is the part where we talk about Gradient!

Teams also screen for AI skills by using Gradient assessments.

The core idea is simple: AI skills should be assessed through realistic work, with a consistent rubric, in an environment that respects candidates and gives hiring teams better signal.

Gradient helps teams evaluate how candidates actually work with AI: how they reason, prompt, iterate, verify, and produce role-relevant output. Instead of relying on self-reported AI usage or invasive personal tool checks, companies can run structured assessments that are fairer, more consistent, and easier to compare across candidates.

That matters because AI fluency is quickly becoming a hiring signal across many roles. But without a thoughtful assessment process, companies risk selecting for confidence, jargon, or tool access instead of actual capability.

The question we want teams to be able to answer is "Can this person use AI to do better work?", which is a harder thing to ask and a much more useful thing to know.

If that sounds interesting or relevant to you, please don't hesitate to reach out.