Small samples, big noise — when a QA difference is real

Quality · ~7 minute read

A team’s QM score moves from 84% to 89%. A coaching pilot of ten agents beats a control group. Inter-rater agreement dips for three weeks running. The question underneath all three is the same: is that real, or is it noise? Most QM functions cannot answer it — and so they diagnose noise, celebrate luck, and miss genuine drift.

Inter-rater agreement is a metric, not an event

Calibration sessions produce a number — the proportion of scoring decisions raters agree on — and most programmes treat it as a periodic report. The advanced move is to treat it as a continuously tracked health metric with SPC applied: plot agreement weekly or monthly, compute control limits from a stable historical period, and flag points outside the limits or run-rule patterns.

What SPC surfaces that periodic reading misses: a single point below the lower limit (a new rater, an interpretation change that hasn’t propagated); a run of seven or eight consecutive points below the mean (drift — the calibration discipline isn’t holding); a steady trend (the most insidious pattern, invisible week to week); and widening variance (raters pulling apart even while the mean holds). One operation watched agreement erode from 87% to 81% over nine months with no single alarming dip — the SPC chart showed a clean run below baseline, the cause was a new rater cohort never cross-calibrated with the old one, and a targeted six-week programme fixed it before it became a crisis.

The right statistic for the right purpose

Raw percent agreement is fine for routine monitoring. For sharper questions, two refinements: Cohen’s kappa corrects for the agreement two raters would reach by chance — on a 3-point scale, roughly a third of agreement is luck — making it the right statistic for cross-team or cross-programme comparison where raw agreement can be inflated by skewed score distributions. Intra-class correlation is the tool when more than two raters score the same calls.

Don’t over-statistic the routine reporting; do bring the sharper tools out for the harder questions. Percent agreement for the weekly chart, kappa for the monthly programme-health view, ICC for the periodic multi-rater deep-dive is the right mix for most operations.

The coaching experiment — QM’s A/B test

The most valuable use of significance testing in QM is testing whether a coaching intervention produced change beyond what would have happened anyway. The design disciplines: pre-specify the hypothesis and the primary metric; define pilot and control groups (random where ethical, matched where not); power-analyse the sample — detecting a 4-point difference typically needs 15–20 agents per group over 6–8 weeks; run with observability so you know the pilot was actually delivered; analyse with difference-in-differences.

Pre-specification is the discipline that prevents p-hacking. Testing the coaching effect on every scorecard item and reporting the ones that came up significant guarantees false findings — with fifteen items at p<0.05, one will be ‘significant’ by chance. Name the primary metric up front; treat secondary metrics as hypotheses for future investigation, not findings.

Confidence intervals over p-values

For a rollout decision, the confidence interval beats the p-value because it carries direction, size and uncertainty in one statement. ‘Pilot scored 5.2pp higher than control, 95% CI [2.1, 8.3]’ lets leadership plan to a range: if even the lower bound clears the operational threshold for action, the decision is clear; if the interval is wide and crosses zero, more data is the right call. ‘p<0.01’ alone says none of that.

And hold statistical and practical significance together. With tens of thousands of scored contacts, a 0.3pp difference between teams is statistically significant and operationally trivial. With a six-agent pilot, ‘no significant effect’ means nothing — the test couldn’t have detected a real one. Underpowered tests that report absence and overpowered tests that report trivia are the twin failures; power analysis up front and an explicit action threshold prevent both.

Communicating decisions, not jargon

The QM lead’s job is translation: design, result, uncertainty, operational interpretation, recommendation — in one paragraph. ‘Six-week experiment, 16 pilot agents against 24 control; resolution clarity lifted 5.2pp with 95% CI [2.1, 8.3]; our rollout threshold is 3pp and even the lower bound clears it; recommend rollout next quarter.’ Leadership needs the decision the statistics supports, not the statistics.

Done well, this turns the QM function into a source of defensible evidence for coaching investment. Done badly, every pilot’s ‘we think it worked’ carries no weight beyond the conviction of the people who delivered it — and the next budget conversation goes accordingly.

The closing principle

Before diagnosing any QM movement, ask whether it would survive an SPC chart; before claiming any coaching effect, ask whether it would survive a controlled comparison. The function that can say ‘5.2pp lift, CI [2.1, 8.3], rollout justified’ earns a hearing the function that says ‘it felt better’ never will.