How Linear Runs Evals on Its AI Agent — Jacob Shumway (Peter Yang Short)

A 44-second clip from Peter Yang’s interview with Jacob Shumway (Linear).

• Evals mostly come from iterating and using the agent: a user uses it in an unexpected way → that case gets added to the eval dataset, and the evals grow over time
• They keep a mix of objective and subjective measures
• Objective = deterministic: “if the user says ‘in progress’, make sure it always adds the status in progress”
• Subjective = LLM-as-judge: “did you structure the description in a good way? did you extract the right information for the title?”

“We build out this data set over time, and then we just have a scoring type of LLM that’s like, ‘did this extract the right information? This is what it should be.’”