Bumble product manager Supriya Subudhi describes an A/B test of AI-generated feedback on dating-profile photos at AI Engineer.

Interest is not adoption

  • Subudhi says 78% of people consulted wanted feedback, but only 10–15% used the feature in the test; the groups and measures are not necessarily directly comparable.
  • Placing prompts in existing photo-upload and profile journeys increased adoption, she says; she gives no post-change figure.

Look beyond the average

  • Profile edits increased overall, but some users received poor advice, such as being told to smile when already smiling.
  • The team examined real outputs and segments, including gender, language and country, and set limits on feedback about traits people cannot control.
  • A positive experiment result does not show that every user receives useful or fair feedback.

Check the economics and the rollout

  • Subudhi says the initial best-model approach cost roughly ten times a simpler one, before evaluation costs.
  • Switching models reduced cost but also reduced the observed impact; the team used evaluations to compare models and prompts until it found a cheaper option with a good result.
  • The US version could not simply ship unchanged elsewhere, she says; involve legal and compliance early and decide whether regional versions are worth their cost.

“A positive result from an A/B test is actually just where the journey begins.” — Supriya Subudhi