Chapter 02
Online evaluations
Twig's AI support agent helps visitors with questions and problems during their stays. For an ordinary cancellation, Twig requires at least seven days' notice before arrival for a full refund. The cancellation policy has an exception for broken essential amenities (heating, cooling, hot water, all the human luxuries): if an amenity can't be repaired promptly, the visitor qualifies for relocation or a refund at any time.
The support agent refuses a full refund, applying the ordinary seven-day notice rule instead of the broken-amenity exception.
Twig's engineers configure online evaluations to flag incorrect support answers without waiting for bug reports.
Check what each score measures
Twig's engineers enable three checks for support responses: a code check, sentiment analysis, and LLM judge. Send the heating complaint, then select each result in the inspector:
A code-based evaluation runs a rule written in code. Twig's engineers use a rule that looks for "7 days" in the support agent's response. The evaluation passes because those words are present, but that doesn't establish whether the agent applied the right refund policy.
An LLM judge uses another model to assess a response against criteria the engineers provide. Twig's engineers ask whether the support agent followed the refund policy, including the exception for broken essential amenities that can't be repaired promptly. The evaluation fails because the response refuses a refund despite that exception.
Give evals facts
Twig's engineers name their LLM judge evaluation “Policy correctness” and give it three things:
- The visitor's request: the heating is broken and can't be repaired.
- The support answer: a full refund requires at least seven days' notice before arrival.
- The policy: a broken essential amenity that can't be repaired promptly qualifies for relocation or a refund at any time.
The judge needs all three to assess this answer. The refusal alone sounds consistent with the ordinary cancellation policy. The visitor's message and the broken-amenity exception explain why that policy doesn't apply.
In PostHog, the engineers choose each generation as the evaluation's scope because this model call contains the request, answer, and policy. They choose LLM with explanation to receive a pass or fail with a written reason. For this answer, the judge explains that the support agent ignored the broken-amenity exception.
Before enabling the evaluation, Twig's engineers review sample results for both ordinary cancellations and heating complaints. They confirm that the judge accepts a refusal when the seven-day rule applies and rejects it when the visitor qualifies for the exception.
Use code evals for exact comparisons
Twig's engineers use a code evaluation to catch recommendations like a two-person studio for a group of four. The rule is simple: the stay's capacity must be at least the requested guest count.
They record requested_guests and stay_capacity, custom properties they've named, and write the comparison in Hog, PostHog's evaluation scripting language.
Change the capacity to see how the same rule handles each sample:
Missing capacity needs a third outcome. Twig's engineers enable allow N/A responses and make the rule return null when either value is missing. PostHog records N/A: there isn't enough information to judge whether the stay fits.
They monitor N/A results alongside passes and failures. A missing capacity must not count as a suitable recommendation.
Apply checks to the right requests
Twig's engineers filter the policy evaluation to the support feature, then choose how much matching traffic to sample. The policy judge makes another model call, so the engineers consider its cost as well as the coverage they need. The wording check and sentiment analysis don't incur that LLM judge cost.
After reviewing the criteria, Twig's engineers use a backfill to apply the policy check to eligible historical requests. They check that those requests still contain the policy and messages needed for a judgment, and review the filters, sampling, and expected cost before starting it.
Design an online eval for your project
Copy this prompt into PostHog AI: