Scored against somebody else’s answer key
Every number on this page comes from vouch.finance’s supervisory kits: their programme, their
transaction stream, their scripted ground truth, and their own scorer.ts reading
it. That is the only reason a figure from it is worth anything.
All four kits were seeded and scored, including the one AMW is worst at. A benchmark published only when it agrees with you is a press release.
| Kit | Payments | Precision | Recall | Recall* |
|---|---|---|---|---|
agent-mandate | 52 | 66.7% | 70.6% | 61.8% |
agent-mandate, seven arena personas | 73 | 54.4% | 100.0% | 51.6% |
kya-licence | 19 | 100.0% | 100.0% | 100.0% |
embedded-supervision | 6 | 66.7% | 100.0% | 100.0% |
disbursement-integrity | 41 | 0.0% | 0.0% | 0.0% |
Latency to detection is 0ms on every run, and it is not an optimisation: the guard evaluates the intent and holds it before value moves, so detection is the transaction rather than a report written about it afterwards. There is nothing to be late for.
Recall* is the column to read
It counts a violation as detected only when a clause about that violation cited it.
The raw figure counts anything the guard held, and a great deal was held on clause 3.5, the
action count, which catches a payment for being the seventeenth of a window rather than because
anything recognised what it was. Fifteen of the arena’s thirty-one detections
are that.
Read that way, the substantive rate on the spending kit is 61.8%, which is exactly what the previous deployment reported. The arithmetic around the mechanism became exact; the mechanism did not get better at recognising fraud, and saying otherwise would be reading a bound nobody asked for as understanding.
Every false positive on the spending kits is one clause
Clause 3.5, which Vouch’s programme does not state at all. The window is summed exactly, so the ledger it is summed from has to be bounded, so the registry refuses a mandate admitting more than sixteen actions to one window. AMW cannot write this programme down without adding a bound to it, and any actor whose week runs past sixteen payments meets a clause nobody asked for.
| Run | Precision | Of which clause 3.5 alone | Setting those aside |
|---|---|---|---|
amw-3 | 66.7% | 12 of 12 | 100.0% |
amw-arena-1 | 54.4% | 21 of 26 | 86.1% |
Both figures are published because they are different faults. One is AMW being wrong about a payment; the other is AMW being unable to express somebody else’s programme. Averaging them hides the second, which is the one worth knowing.
The zero is the most useful number here
disbursement-integrity scores 0.0% precision and 0.0% recall, and
it would not have scored better on a better day. Its nine violations are six
mule_pattern (six different beneficiaries converging on one flagged merchant, each
spending inside its own entitlement) and three synthetic_identity, a
duplicate-identity beneficiary transacting normally.
Neither is a thing AMW does. A mule pattern only exists across instruments, and cross-agent aggregate exposure is the limitation the console states about itself in its own words: the guard bounds each instrument and every mandate above it; it does not bound the sum. A synthetic identity is a check at enrolment, and AMW is asked its question after an owner has already signed a mandate naming an agent.
Its six false positives are worth reading rather than dismissing. Every one is a quota hold on a beneficiary who spends 566.56 against a 300 entitlement, and Vouch’s own engine refused ten of the same twelve payments. The answer key is not wrong. It labels the fraud it scripted, and a quota overrun is not in its taxonomy for this kit. But precision measured against it here does not mean what it means on the other kits.
This is the boundary of the product, drawn by somebody else. It is a better thing to have than a fifth good score.
The identity lifecycle, reproduced rather than scored
kya-licence scores 100% on both axes over five counterparty violations, all held on
clause 2.3 before value moved. But the half of that kit worth having is not scored at all: three
of its eight steps are acts on the mandate rather than payments, and the answer key
labels transactions. Inventing a number for them would be worse than leaving them out.
| The kit’s step | What Vouch answered | What AMW answers |
|---|---|---|
| Delegate a grandchild mandate | Parent voucher does not permit delegation | TooDeep: depth is derived from the parent at registration and never asserted by the signer |
| Revoke the child mid-stream | allowed; a revocation is a legitimate act | revoke, owner only. Descendants are deliberately untouched: a revoked ancestor fails the guard’s chain walk |
| Reuse the revoked mandate | Parent voucher is not active | REVOKED, a tier-1 flag, held before value moves, so the attempt is in the intent log |
What Vouch refuses in a policy engine it can amend, AMW refuses in a contract nobody can. Each
row is put to the contract in packages/vouch/src/lifecycle.test.ts rather than
described here and believed.
Two things to distrust in this page
embedded-supervision is six payments and two violations, and the kit’s own
on-chain contracts did not deploy during seeding: its actors fell back to database
enrolment. That run exercises the payment path and not the ledger reconciliation the track is
named for, so its 100% should not be read as evidence about rail-agnosticism.
And the figures published on 1 September were scored against contracts that no longer exist:
today’s registry refuses the mandates that mapping registers, with
ActionsOutOfRange. A test pinned to the recorded flag count is what said so, rather
than a benchmark quietly describing a deployment nobody runs. The artefacts for every run
(labels, payments, flags, the platform’s own decisions and the arena’s report) are in
packages/vouch/runs/.