AI Evaluation & Reliability
How do we know that an agentic system keeps being correct, and calibrated, as data changes?
Research question
How do we know that an agentic system keeps being correct, and calibrated, as data changes?
Problem definition
A system can get worse in silence when its inputs drift. A confidence of 90% that is right only 70% of the time makes every threshold built on it wrong.
Scientific motivation
Reliability is measurable: accuracy, calibration error and the rate of cases sent to people are numbers that can be tracked, compared with a baseline and used to trigger review.
Mathematical foundation
Predictions are grouped into M confidence bins B_m. ECE is the average gap between accuracy and mean confidence, weighted by bin size.
Proposed methodology
Build a test set that resembles production.
Measure accuracy, calibration and review rate by task.
Compare with a baseline before releasing a change.
Monitor calibration, data drift and human feedback after release.
System architecture
Test set
System run
Metrics
Baseline comparison
Release decision
Monitors
Experiments
No published experiments. The Agent Evaluation lab simulates an agent with adjustable overconfidence and shows its reliability diagram, ECE and cost-optimal threshold.
Results
No original research results are claimed. This area is presented as a technical perspective with a computable demonstration on synthetic data.
Limitations
- ECE depends on how the bins are chosen.
- A test set is not production.
- Metrics cannot detect a failure nobody anticipated.
References
Background reading by other authors. These are not Menlo Tech publications, and no Menlo Tech paper is cited because none has been published.
- Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. "On Calibration of Modern Neural Networks." ICML, 2017.