Confidence Calibration: The Model Says 0.9 — What Does That Actually Mean?

A model reports 0.9 confidence on an extraction. Does that mean 90% accuracy? Not necessarily. One team set their review threshold at 0.85 based on the assumption that “0.85 confidence means approximately 85% accuracy.” An audit found that 40% of total errors were in extractions scored above 0.85 — the model was confidently wrong, and the threshold let those errors through.

Confidence scores are useful for routing, but only after empirical calibration. The number the model reports and the accuracy it actually achieves at that number can differ significantly.

Calibration: Comparing Claims to Reality

Calibration measures whether reported confidence matches observed accuracy. A well-calibrated model achieving 90% accuracy at reported 0.9 confidence is useful. A model achieving 75% accuracy at reported 0.9 confidence is overconfident and dangerous for automated routing.

A team evaluated their model’s calibration against a labeled validation set:

The model is reasonably calibrated in the 0.5-0.9 range. But in its highest confidence band (0.9+), it achieves only 82% accuracy — meaning nearly 1 in 5 of its most confident extractions are wrong. A threshold set at 0.8 (assuming “80% is good enough”) lets these overconfident items through.

How to Calibrate

Before deploying confidence-based routing:

1. Create a labeled validation set — human-verified correct values for a representative sample of extractions
2. Bin by confidence — group extractions by their reported confidence ranges
3. Measure actual accuracy per bin — what fraction are actually correct in each range?
4. Compare reported vs actual — identify where the model is overconfident or underconfident
5. Set thresholds based on observed accuracy — not on the model’s claimed confidence

There is no universal “0.8 is the right threshold.” Each model, field type, and document source needs its own validation. The threshold should be set where actual accuracy drops below the acceptable level for that specific use case.

Per-Category Calibration

The model may be well-calibrated for some categories and overconfident for others. One system found warranty_period fields averaged 0.88 confidence but had a 25% error rate, while other fields at similar confidence had only 3% error rates.

A single global threshold treats all fields equally. But if warranty extractions are overconfident, the global threshold lets their errors through. Per-field or per-category calibration sets different thresholds where different fields cross the accuracy bar:

• Invoice number: threshold 0.8 (well-calibrated, 0.8 confidence ≈ 82% accuracy)
• Warranty period: threshold 0.93 (overconfident, need higher reported confidence to actually reach acceptable accuracy)
• Legal terms: threshold 0.90 (moderate overconfidence on complex text)

This matches the routing to each field’s actual reliability profile.

Calibration Drift

Calibration is not permanent. When the data distribution changes, the accuracy-confidence relationship shifts.

One team calibrated their system and achieved excellent results: 0.85+ confidence corresponded to 91% actual accuracy. Six months later, a re-calibration check showed 0.85+ confidence now corresponded to only 76% accuracy. The cause: 25% of messages now came from a new mobile app with different formatting that the model handled less reliably.

The review queue volume was unchanged (same threshold, same proportion routed), but the quality of auto-accepted extractions had silently degraded. Without periodic re-calibration, the team would not have known.

Re-calibrate on a regular schedule (quarterly is a reasonable default) and after significant data distribution changes (new sources, format updates, customer channel additions).

Override Rules: When Confidence Is Not Enough

Confidence-based routing has a blind spot: the model is sometimes confidently wrong, and no threshold can catch errors the model does not know it is making.

Two situations require override rules that bypass confidence entirely:

High-impact categories. Security vulnerability findings should always receive human expert review regardless of confidence. The cost of a missed critical vulnerability far exceeds the review cost. If the model is known to be overconfident on security findings, confidence-based routing for that category is not just suboptimal — it is actively dangerous.

Ambiguous source material. Faded scans, handwritten amendments, contradictory specifications — when the source itself is ambiguous, the model may report high confidence on its interpretation while another interpretation is equally valid. Override rules that detect source ambiguity (format markers, specification quality indicators) route these to review regardless of extraction confidence.

Impact-Based Prioritization

When the review queue exceeds reviewer capacity (500 items/day, team capacity 150), confidence alone is the wrong sorting key. A low-confidence customer name extraction and a low-confidence refund amount extraction are not equally important to review.

Impact-based prioritization within the review queue:

1. High impact + low confidence → review first (refund amounts, legal terms)
2. High impact + moderate confidence → review next (overconfidence risk on critical fields)
3. Low impact + low confidence → review if capacity allows (customer names, formatting)

This allocates scarce reviewer time to where errors are most costly. FIFO processing treats all fields equally, wasting capacity on low-impact items while high-impact errors wait.

The Routing Matrix

For systems with multiple dimensions (finding category, severity, confidence), a routing matrix optimally allocates limited review resources:

This matrix integrates calibration (per-category thresholds), overrides (critical security always reviewed), and resource allocation (lower severity gets lower priority). A single global threshold cannot encode these distinctions.

The Calibration Lifecycle

Confidence calibration is not a one-time setup:

1. Initial calibration — before deployment, validate against labeled data
2. Threshold setting — per-category thresholds based on empirical accuracy
3. Override rules — for categories where confidence is unreliable
4. Monitoring — track accuracy at each confidence level in production
5. Re-calibration — quarterly and after distribution shifts
6. Threshold adjustment — update when calibration data shows drift

Skipping step 1 (deploying with uncalibrated scores) is the most common and most dangerous anti-pattern. Everything downstream — thresholds, routing, review prioritization — depends on confidence scores actually meaning something. Without calibration, they are just numbers.

One-liner: Calibrate confidence scores against labeled data before trusting them for routing — the model’s reported 0.9 might only be 82% accurate, and only empirical validation reveals the truth.