Why calibration matters
Every threshold built on a confidence, such as the one derived in the note on cost-based thresholds, assumes that the confidence means what it says. When a model reports 90% and is right 70% of the time, the decision rule is wrong even though nothing in the code has changed.
Measuring it
Group predictions into M confidence bins. In each bin compare the fraction that were correct (accuracy) with the mean reported confidence. The expected calibration error averages the gaps, weighted by bin size:
A worked example
Illustrative numbers. Of n = 100 predictions, 60 fall in a bin with mean confidence 0.55 and accuracy 0.50 (gap 0.05), and 40 fall in a bin with mean confidence 0.95 and accuracy 0.80 (gap 0.15). Then ECE = 0.6 × 0.05 + 0.4 × 0.15 = 0.09.
A proposed monitoring design
This section is a proposal, not a description of a deployed system. Three monitors run after release: calibration (ECE over a rolling window), drift (a change in the distribution of inputs) and human feedback (the rate at which reviewers correct the system). Each has a threshold. Crossing it triggers a review of the model and of the decision threshold, and it is recorded as an event.
Limits
- ECE depends on how the bins are chosen and needs enough samples per bin.
- Averaging hides large errors in rare bins.
- A held-out test set is not production; monitors must run on live data.
Background reading
Works by other authors that treat the underlying methods. They are not Menlo Tech publications.
- Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. "On Calibration of Modern Neural Networks." ICML, 2017.
Related
Labs
Case studies