Vishal Srivastava and Tanmay Sah of New York Life discuss system-level evaluation in this 20-minute AI Engineer talk.
Study the failure chain
- Their review covers 1,368 publicly reported AI incidents, according to the uploader description.
- The framework separates technical weaknesses, deployment controls and business consequences.
- They report that 13 sub-dimensions describe about 98% of incidents, with roughly six involved per incident on average.
- These are retrospective classifications of reported incidents, not estimates of all AI failures or future prevention rates.
Test the controls around the model
- Technical checks cover data, design, output validity, fairness, failure modes and explainability.
- Deployment checks cover decision authority, effective human review, exposure and monitoring.
- Business impact includes financial costs, legal consequences and the scale of affected users.
- Weak human review and harm detection show stronger associations with harm in their analysis than frequency alone suggests.
Challenge the claim, then test the fix
- Risk depends on failure frequency, whether an error reaches a real decision, whether it escapes detection and its consequences.
- Test retrieval, broken tools, permissions, escalation and recovery alongside the model’s answers.
- Check the reliability and coverage of any model that generates tests or grades outcomes.
- Map reliable, uncertain and high-risk operating regions; limit deployment to supported uses.
The airline example
- A chatbot’s incorrect bereavement-fare advice illustrates an error reaching a customer and creating organizational liability.
- Their proposed loop challenges policy answers, adds policy checks or escalation, then tests those controls again.
- This is a retrospective application of the framework, without evidence that these particular tests would catch every such failure.
“Evaluate the full AI system based on risk, not just one component.” — Srivastava, 19:36