Test suites

A test suite is a list of cases for one assistant. For each case, an AI caller plays the part you describe and talks to the assistant (as text, so runs are fast). A grader then reads the conversation and decides whether it met your criteria.

#A case

nametext

A short label, such as "New patient books a cleaning".

personatext

Who the caller is and how they behave.

goaltext

What they want from the call.

criteriatext

What a successful call looks like.

max_turns2 to 20default 8

How long the conversation may run.

Name:     Emergency after hours
Persona:  Anxious parent. Their child knocked out a front tooth an hour ago. Speaks fast.
Goal:     See a dentist as soon as possible.
Criteria: Tells them to keep the tooth in milk, offers the first emergency slot,
          and books it with the caller's name and number.

#How tools behave in tests

The assistant gets its real prompt and tool definitions, but tools are simulated, so nothing is booked, texted or sent. Availability shows a few free times, and bookings and messages report success. Knowledge search is real and read-only. Switch on real tools to let your own API, code and server tools run for real, which is useful for read-only lookups.

#Running

A suite holds up to 30 cases. Run it after every change to the instructions. Each run shows the full conversation of every case and why it passed or failed.