Hamel Husain explains how to inspect AI-product failures when sensitive customer information restricts access, in an 81-second clip on his channel. YouTube reports embedding disabled for this video; use the watch link above.
Work down the access ladder
- Get permission for real samples: Ask for approved test users, test datasets, or a limited user subset. Partial authorized access beats no inspection.
- Partner with authorized experts: If only doctors or lawyers may inspect the traces, make the product easy for them to verify rather than routing sensitive data to unauthorized developers.
- Redact traces: Remove sensitive details before reviewing them. Whether a library or LLM does the redaction, rigorously test that private information actually disappears.
- Use synthetic data only as a last resort: It may reveal obvious problems, but is unlikely to expose the subtle errors that occur with real users.
“There is no replacement for looking at real data.”