Scoring
Gradient’s scoring engine uses an LLM-as-judge approach to evaluate candidate submissions across multiple dimensions. Our scoring framework is informed by Anthropic’s AI Fluency Index, which identifies the core competencies that distinguish effective AI-augmented knowledge workers. The result is a detailed score breakdown with percentile rankings and suggested follow-up interview questions.Scoring categories
The default rubric (v5) evaluates candidates across five categories. The overall score out of 100 is a weighted blend of the five, using these fixed weights:Category weights are fixed by the rubric version. Gradient sets the sub-criteria inside each category when it builds the assessment (see Scoring Rubrics).
AI Fluency is managed centrally by Gradient. It is auto-graded from session behavior, always locked, and not manager-calibratable, so this dimension stays consistent across every organization and assessment.
How scoring works
1
Trigger
After a candidate submits, an admin triggers scoring via the dashboard or the Scoring API. Scoring can also be re-run on already-scored sessions.
2
Analysis
The scoring engine runs multiple analysis modules in parallel. Each module examines different aspects of the submission: the deliverable itself, the conversation transcript, event patterns, and the AI configuration snapshot.
3
Scoring
Each sub-criterion receives a score using one of these methods:
- Deterministic: Ground-truth checks, used for Correctness (are the required facts and elements present and right?)
- LLM Judge: An AI evaluator assesses quality against rubric criteria (used for Deliverable Quality, Reflection, and Prioritized Skills)
- Event Analysis: Automated analysis of behavioral patterns (for example, did they verify a figure before using it?)
- Hybrid: Combines approaches. AI Fluency is graded this way, centrally, by Gradient
4
Percentiles
Scores are benchmarked against one baseline: other candidates who sat the same assessment. The pool is scoped to a single assessment, so a candidate is never ranked against another organization’s candidates. A percentile appears once at least five other candidates have completed that assessment.
Follow-up questions
The scoring engine also generates suggested follow-up questions for the hiring manager. These are specific, evidence-based questions tied to things the candidate did (or didn’t do) during the assessment. For example, if a candidate accepted incorrect data from the AI without verification, the engine might suggest:“I noticed you included a revenue figure that doesn’t match the source document. Walk me through your verification process for AI-generated data.”Each question includes:
- The question itself
- Rationale: Why this question matters (what the scoring engine observed)
- What to look for: Strong vs. weak answers
Calibration
Gradient learns your standards over time. Reviewers grade candidates on each sub-criterion using its 0-4 anchored scale (the same ladder the AI grades against). When a reviewer’s grade disagrees with the AI, that disagreement feeds calibration, which sharpens future scoring for the assessment. Calibration routes each item to the right channel:- Durable and process items (Prioritized Skills, Reflection) refine the sub-criterion’s 0-4 anchor descriptors, so the ladder the judge reads captures what “good” means for this role. See anchored scales.
- Deliverable Quality tunes a learned preference profile rather than anchor text, because that category is scored by comparison, not by a written ladder.
Admin review
After automated scoring, admins can:- Review the detailed score breakdown per category and sub-criterion
- Replay the candidate’s session (every action, message, and edit)
- Adjust individual category scores with the Override Score API
- Add review notes
- Release feedback to the candidate (immediately or after a configurable delay)