How accurate is AI call scoring, and can a rep challenge the score?
Yes, a rep can challenge the score, and should be able to. We found no independent, published accuracy figure for AI scoring of a sales conversation. In the closest study, adding an old coaching note to the prompt changed 16.4% of yes-or-no answers on average, across 18 models and 3,000 contact center transcripts. In Spellit Actions, a rep can send their disagreement with a score from inside the product.
The usual pitch for AI scoring is consistency. A vendor blog post from September 2025 puts it plainly: the same call will receive the same score every time, regardless of when it's evaluated. Human reviewers drift, the machine does not, so the argument goes, and the machine is the fairer judge.
The trouble is that consistency on an identical input is not what a rep is worried about. In normal use, nobody scores the same transcript twice. What the rep wants to know is whether the verdict came from what they said on the call, or from something else the scorer had in front of it. On that question there is now data, and it does not support the pitch.
No published figure tells you how accurate AI scoring of sales calls is
We could not find an independent number to quote. We looked for one before writing this, and what exists falls into two groups: vendor percentages with no sample size, period or method attached, and research on neighbouring tasks that measures something narrower than a sales call scorecard.
The neighbouring research is consistent on one point. Accuracy is a property of the individual line on the form, not of the scorer as a whole. A preprint on judging voice agents, revised in September 2026, found that model judges were usable on some conversation metrics and unreliable on others, and that reliability changed between the telecom and retail data it tested. We made the same argument for human scorers in the piece on calibration: lines that describe an event on the recording agree far better than lines that ask for a judgment.
So "how accurate is the AI" has no single answer, and a vendor who gives you one is averaging over the lines where it is reliable and the lines where it is not. The useful question is line by line, on your own conversations. Nobody outside your company can answer it from a brochure. Name the line your team argues about most, and we will score it on a sample of your recordings.
The same transcript gets a different verdict depending on what the scorer was told
A verdict can flip without a word of the call changing. The most direct evidence comes from a February 2026 study led by researchers at Observe.AI, a contact center software company. They ran 18 language models over 3,000 real contact center transcripts, asking 30 yes-or-no quality questions of the kind found on any scorecard, such as whether the agent acknowledged the customer's concern. Then they changed one detail at a time and counted how often the yes-or-no answer flipped.
| What was changed before scoring | Share of answers that flipped, average of 18 models |
|---|---|
| A short note from previous coaching, such as "Needs to improve empathy" | 16.44% |
| Metadata about the context of the call | 10.08% |
| Ethnicity of the agent, signalled through cues in the conversation | 9.71% |
| A profile of the customer | 9.64% |
| A summary of the agent's past quality scores | 9.00% |
| How polite the agent sounded | 6.93% |
| Gender of the agent | 6.78% |
The transcript of the call itself did not change in the top row. Only the note in front of it did. The best model in the study still flipped 5.41% of answers on average across all the changes tested. The worst flipped 12.95%. Telling the models explicitly to be fair moved the rate by a few points at most, and for one model it made things slightly worse.
The authors work for a company that sells AI quality assurance, and they published a result that cuts against the usual pitch for it. They also report that the fairest models were not simply the most accurate ones. A scorer can be right on average and still be moved by what it was shown.
The operational rule follows directly. Keep the rep's history away from the scorer. Previous scores, coaching notes and manager comments belong in the coaching conversation, never in the prompt that judges the next call. And when a rep challenges a line, the person deciding should look at the recording before looking at the rep's record, because the record is the thing most likely to have bent the verdict.
The weekly coaching piece made the short version of the case: a rep has to be able to challenge a score, and a human has to be able to overturn it. What it leaves out is how the challenge should actually work.
A challenge needs five fields, and the rep should not have to write an essay
A challenge that takes twenty minutes to file will not get filed, and an accuracy measurement nobody feeds is worthless. The form should ask for the minimum a decider needs to settle the question against the recording, and nothing about the rep's feelings or record.
| Field | Why it is there |
|---|---|
| Which call and which line | A challenge to "my score" cannot be checked. A challenge to one line on one call can |
| The verdict the rep thinks is right | Turns a complaint into a yes-or-no question the decider can answer |
| Where in the recording | A timestamp lets the decider listen to the thirty seconds that matter instead of the whole call |
| Why, in a sentence or two | Required. Without it the decider is guessing what the rep heard |
| A deadline | A challenge that waits a month is decided after the coaching conversation it should have changed |
In Spellit Actions, a rep can send their disagreement with a score from inside the product. Whatever tool you use, the fields above are what the person deciding needs in front of them.
Who decides matters as much as what is filed. The person needs the authority to change the result, and ideally not a stake in the scorecard being right. A manager deciding challenges against a form they wrote is marking their own homework. A default worth starting from: the manager decides, and the rep can send any decision up one level, once. If nobody in your company stands outside the form, send us a few disputed calls and we will rule on each against the recording, with the reason written down.
The overturn rate is the accuracy number you were looking for
Count how often challenges succeed, per line. That share, measured on your own calls and your own form, is the closest thing to an accuracy figure that exists for your setup. A line that keeps getting overturned is usually a badly worded line, as the review process piece argued, and the fix is to rewrite it.
The number has a blind spot, and it is a big one. Challenges come from lines that went against the rep. Nobody files a challenge against a pass they did not deserve. So the challenge log only catches errors that went against the rep, and a scorer that is too generous on some line will look perfect in it.
| What the log shows | What it means | What to do |
|---|---|---|
| A line overturned again and again | The line or the model misreads something common on your calls | Rewrite the line, then watch whether challenges drop |
| A line challenged often but almost always upheld | Reps and the form disagree about what good looks like | Talk to the team before touching the form |
| A line never challenged | Either it is accurate, or it errs in the rep's favour | Spot-check a handful of passes against the recording |
| No challenges at all from one team | Nobody believes filing changes anything | Ask why before you read it as accuracy |
A default worth starting from: once a month, take the two most challenged lines and pull five unchallenged passes for each, then check them against the recordings yourself. That covers the direction the log cannot see. Lines where the AI cannot be certain are a separate case: a blank is not a pass and not a fail, and it should not be challengeable as either.
What to do next
Open your scoring setup and check one thing first: does the prompt or the scorer see anything about the rep beyond this call? Previous scores, coaching notes, a tenure field, a team label. If it does, take it out before you do anything else. The largest effect in the study came from a single coaching note.
Then write down, in one sentence each, who decides a challenge, by when, and what they have to write. Run it for a month and read the log line by line, remembering which direction it cannot see.
The challenge log will tell you where your scorer is wrong against the rep. It will not tell you where it is wrong in the rep's favour. Send us both kinds of call and we will check the half nobody complains about.
- We could not find a published accuracy figure for AI scoring of sales calls with a sample, a period and a method behind it.
- The same transcript can get a different verdict depending on what the model was told about the rep beforehand.
- A challenge log is the only accuracy measurement you will get on your own calls. It only catches errors in one direction.
- A decision on a challenge with no written reason gives the rep nothing to check, so don't let anyone save one.
A live demo of our products and a real conversation about the growth problem you need to solve.
Book a demo