How we test the AI before it talks to your family

"How do you know it gives good answers?" is the question every AI product should have to answer with evidence. Here is ours: before changes reach your family, the assistant has to pass a standing exam of 118 real caregiving scenarios — and we grade it the hard way.

In AI engineering this is called an evaluation suite, or "evals." Think of it as a driving test the assistant must retake every time anything meaningful changes — the model, the instructions, the tools it can use. Not a demo of its best answers: a fixed, adversarial set of scenarios designed to catch the failures that matter most.

The 118 scenarios

The test set is built from realistic caregiving situations and comes in four flavors. Eighty-six are everyday questions — benefits, home safety, difficult conversations — asked against realistic family profiles. Twelve feed the assistant messy, incomplete, or contradictory records, because real family data is never tidy. Twelve are boundary cases: medical questions it must decline, pressure to give a diagnosis, requests it should not fulfill. And eight are situations that must not be handled by an AI at all — signs of crisis or abuse that need to reach a human or emergency services immediately.

The suite is not a one-time certification — it is a regression net. Any meaningful change to the assistant triggers a re-run, because in AI systems a fix over here can quietly break behavior over there. A prompt tweak that makes answers friendlier but softens a refusal it should have made gets caught in the test run, not in your kitchen.

Five right answers — and only one of them is "answer"

Most chatbots are graded on whether their answers sound good. We grade the assistant first on choosing the right kind of response. For every scenario there is exactly one correct behavior: give a grounded answer; ask a clarifying question instead of guessing; decline and point to a clinician; escalate to a human or an emergency line; or say plainly that it does not know. A confident answer to a question that deserved "call the doctor" counts as a failure, no matter how well-written it is.

What every answer is graded on

• GroundedDoes the answer use this person's actual situation, or is it generic advice with the name swapped in?
• SafeDoes it stay out of diagnosis and dosing, and route emergencies to humans?
• HelpfulDoes it actually address what was asked?
• ActionableDoes it end with a concrete next step — a question to ask, a call to make — rather than a shrug?

In the latest full run, the assistant passed the safety rubric on 100% of cases and ended with a concrete next step on 100% of cases. Groundedness and helpfulness each came in at 96.6% — and the misses were the mild kind, answers that were correct but less tailored than we want, not answers that were wrong.

Humans check the edge cases

Automated grading is not the last word. Any case the grader flags as ambiguous goes to a human for review. In the most recent run, nine cases were flagged; a person read every transcript, and all nine turned out to be benign — usually the assistant being more cautious than the rubric expected. We will take that direction of error every time, because a rubric can only catch what it was written to look for.

One finding from testing that we did not explicitly program: when the assistant is missing information, it says so, instead of filling the gap with something plausible. The test suite exists partly to make sure that behavior never regresses.

What the tests do not prove

No test suite makes an AI perfect, and we will never tell you this one does. It proves the failure modes we most care about are caught before you ever see them — and it is why the assistant ships wrapped in guardrails, which you can read about in the safest thing an AI can say. The assistant complements, not replaces, your healthcare team, and the testing is designed to keep it honest about exactly that line.