Pilot protocol

How the pilot measures change

Status: planned · 4-week design

Sample
2 teams
Duration
4 weeks
Roles
4
KPIs
5
Design
Baseline → follow-up
Pilot measurement protocol. No results are recorded yet.
ID KPI Formula Unit Data source Related guideline Baseline Follow-up
K1 Accountability names the reviewer and the escalation route. named reviewer AND route ÷ participants % of participants scenario decision Name the reviewer and the escalation route. Wk 1 Wk 4
K2 Knowledge spots the misleading output and names the check. spots misleading output AND names the check ÷ participants % of participants written explanation Check the source before trusting the answer. Wk 1 Wk 4
K3 Ownership a workflow with named owners and a tested recovery route. named owners AND tested recovery ÷ 2 workflows x/2 workflows workflow note and rehearsal record Name an owner at each critical handoff, and rehearse recovery. Wk 1 Wk 3–4
K4 Process the guardrail is applied, and gaps are documented. guardrail applied AND gaps documented ÷ 2 teams x/2 teams work output and adaptation log Use the override guardrail and record the reason. Wk 1 Wks 2–4
K5 Library logged gaps are reviewed for the shared library. adaptations reviewed for the core ÷ 2 teams x/2 teams adaptation log Log where the shared guidance was unclear. Wk 1 Wks 2–4

Confidence (1–5 self-rating): tracked alongside each decision, not a success measure.

  1. Wk 1 Observe the work ◆ K1–K5 baseline
  2. Wk 2 Build the path
  3. Wk 3 Rehearse a failure ◆ K3
  4. Wk 4 Compare the teams ◆ K1–K5 follow-up

Results are reported as counts and percentages, e.g. ‘4 of 6’. Team-level KPIs are reported out of 2. Each follow-up uses a comparable scenario with new case details.

The team, the toolkit, the product

This pilot measures the team. The shared library holds the guidelines. How the AI product behaves with a person is a separate design question.

How this pilot relates to the shared library and to the HAX Toolkit.
The team The toolkit The product
Question Can people review, own and correct AI together? Do the shared guidelines work, and where must they change? How should the AI behave with a person?
Where it lives This pilot, on each team’s real work. The shared library: guidelines, modules, workbooks and examples. The product itself. Microsoft’s HAX Toolkit (external site) publishes guidelines for that.
Measured here K1–K3 K4 and K5. Practical examples in the library draw on the HAX Toolkit. They are not a copy of it. No. Product behaviour is not a pilot KPI.

Pilot measures

How the pilot would be measured

Five measures, the same across the company, tested on each team's real work. Together they show whether governance is moving from paper into practice.

Planned · baseline week 1, follow-up week 4

One core, flexible for every role and team

Company core 5 KPIs, the shared library and the shared guardrails. Same everywhere, so results can be compared.

  • Developer
  • UX/UI Designer
  • Product Owner
  • Caseworker

Role layer. Each role gets scenarios from its own work.

Team layer. Each team brings one real workflow and adapts the guardrails locally.

  1. Company core. 5 KPIs, the shared library and the shared guardrails. Same everywhere, so results can be compared.
  2. Role layer. Developer UX/UI Designer Product Owner Caseworker Each role gets scenarios from its own work.
  3. Team layer. Each team brings one real workflow and adapts the guardrails locally.

Local adaptations and gaps flow back into the core.

What stays fixed: the five measures, how they're counted, the shared guidelines.

What flexes: the scenarios, the examples, the team's own workflow and agreement.

The five measures

K1

Accountability

Participant Proposed

What it shows People know who reviews an AI output and where to escalate.

Formula participants naming both reviewer and escalation routeparticipants completing the scenario

Example scenario

The Mandated Assistant. The AI drafts “ineligible” while the check is still pending.

The support lead reviews any non-final status; escalate to the product owner the same day.

Someone in compliance should look at it. No named reviewer, no route.

Governance signal Shows whether accountability is known in practice, not just written in a RACI chart.

Evidence
written decision in the scenario
Timing
week 1 and week 4
Reported as
count and %, e.g. “4 of 6” (format example)

Flex Each role gets a scenario from its own work; the core question stays the same.

K2

Knowledge

Participant Proposed

What it shows People can spot a misleading AI output and know which check or expert it needs.

Formula participants who identify the misleading output and name the check or expertparticipants completing the scenario

Example scenario

Innovation First, Risks Later. An assistant trained on synthetic data gives a confident answer about a rule that changed last month.

The source is out of date. Check the policy version and ask the policy specialist.

This looks wrong. Spots it, but no check is named.

Governance signal Shows whether judgement is spread across roles, or still sits with one specialist.

Evidence
written explanation
Timing
week 1 and week 4
Reported as
count and %

Flex A developer's scenario uses logs and tests; a caseworker's uses a citizen's call.

K3

Ownership

Team / workflow Proposed

What it shows Critical handoffs have named owners, and the recovery route has been rehearsed.

Formula workflows with a named owner at each critical handoff and a rehearsed recovery routeworkflows in the pilot

Example scenario

Last-Minute Changes, No Time for Risk. A late change breaks how cases are routed.

The workflow map names an owner at each handoff, and the team rehearsed the fix in week 3.

Owners are named on paper, but the recovery was never tried.

Governance signal Shows whether governance survives real change, not just the happy path.

Evidence
workflow map plus the week 3 rehearsal record
Timing
week 1, then weeks 3–4
Reported as
x of 2

Flex Each team maps its own workflow; the definition of “critical handoff” is agreed locally.

K4

Process

Team Proposed

What it shows Teams apply a shared guardrail in real work and record where it needs adapting.

Formula teams that apply the relevant guardrail correctly and log where the guidance was unclearteams in the pilot

Example scenario

The Case That Doesn't Fit. The caseworker needs to override the assistant.

The override guardrail is used, the reason is recorded, and the team notes “escalation time not defined” for the library.

The override happens informally, by phone, with no record.

Governance signal Shows whether the shared guidelines work in practice, and where the core library must improve.

Evidence
the work output plus the adaptation log
Timing
weeks 1–4
Reported as
x of 2

Flex Teams adapt the guardrail locally; every adaptation is logged and reviewed for the core.

K5

Library

Team Proposed

What it shows Logged gaps are reviewed for the shared library.

Formula adaptations reviewed for the coreteams in the pilot

Example scenario

The Case That Doesn't Fit. The team has noted that escalation time is not defined.

The note is reviewed for the shared library.

The gap stays with the team and never reaches the library.

Governance signal Shows whether local adaptations can improve the shared guidelines.

Evidence
adaptation log
Timing
weeks 1–4
Reported as
x of 2

Flex Each team logs its own gap. The core decides what becomes shared.

What leadership sees

Example layout, no data yet

People who know the reviewer and route (K1)

—

People who can spot a misleading output (K2)

—

Workflows with owners and a tested recovery (K3) x/2

—

Teams using guardrails and logging gaps (K4) x/2

—

Library improvements proposed by teams (K5)

—

The same view works across the company, because the core measures don't change, while each team's scenarios do.

What the results would change

A pattern, and the decision it would lead to. No thresholds.
PatternDecision
Both teams struggle with the same item Update the core library.
One team struggles because of its own workflow Allow local flexibility.
Nobody can name an owner Clarify ownership. It isn't a training problem.
Teams adapt a guardrail in the same way Make the adaptation part of the core.
Confidence rises but decisions don't improve Revise the practice task.

Confidence is rated 1–5 alongside each decision and reported separately. It is not a success measure. Follow-up scenarios use new case details so people can't repeat a memorised answer.