Deciding what counts as working, before anything is switched on

Pick the thing you were trying to change, not the thing that is easy to count

If the problem was that three people get interrupted all day, the measure is interruptions to those three, counted for a week before and a week after. If it was that enquiries arriving at night went unanswered until Monday, the measure is time-to-first-reply on overnight enquiries. Conversation volume answers neither, and it is the number that will be offered. Write the measure down before signing anything, because afterwards the available data decides it for you.

Take the baseline before launch, or you have no measurement

This is the step almost everyone skips and it cannot be recovered later. A week of manual counting before anything changes - tally marks on a sheet is sufficient - converts every subsequent number into evidence. Without it, the only available comparison is against an impression of how things used to be, and impressions move to fit whatever was just installed.

A metric that cannot get worse is not telling you anything

Resolution rate, satisfaction scores on answered conversations and deflection percentages share a property: they are computed over the interactions the assistant handled, so they improve when it handles fewer hard ones. Pair any such figure with a count of what went to a person and what went unanswered entirely. If the vendor cannot show you the second number, the first one is decoration.

Read the transcripts, in a sample, on a schedule

Twenty conversations read properly tell you more than any dashboard, and they are the only way to find the answers that were confidently wrong - those score as resolved. Put it in someone's calendar monthly rather than relying on curiosity. Organisations that do this routinely find that the most valuable output was never the automation but the list of questions nobody had documented.