AK
Aisha Khanpro
1 month ago
jev said approve @ 0.93 on a refund that would have double-charged
Jev
System One model for typed decisions with calibrated confidence
Wired Jev into our refunds review queue Tuesday. Typed Decision looks clean — approve | escalate | deny plus a confidence float.
On a Stripe partial refund that already had a credit note, it returned approve at 0.93. Acceptance criteria said never auto-approve if credit_note_id is set. The schema had the field. The model just… didn’t care.
Pulled it off the auto path. Still useful for escalate triage. Anyone actually thresholding on that confidence number in prod?
5 comments
Join the discussion
Log in to comment.
Did you log the score against a held-out set? Curious what your false-approve rate looks like above 0.9 — I would not trust a single float without a calibration plot on your own refunds.
same. I had it on dispute routing for about a week. Confidence looked pretty until a R$180 chargeback slipped through at 0.88. now I only use it to sort the human queue, never to skip it.
same. confidence sorts the Slack queue, humans still click. had one at 0.91 that would have double-issued a £40 credit note on an already-refunded invoice.
pulled auto-approve after that. triage only — the float is not a green light.
if credit_note_id is already in the schema and it still returns approve @ 0.93, that is not a threshold bug — the model is ignoring the constraint.
we treat anything ≥0.8 as queue-sort only. never skip. payment + refund paths stay on the agent kill-switch list here; green confidence lied twice last month on Stripe partials.
Agree on the kill-switch list. We had Jev at 0.91 approve a partial refund that already carried
credit_note_idin staging — Stripe would have issued a second note for ₦12,400.Confidence sorts our Linear triage now. Auto path stays off until we have a held-out set that survives a week without a double-charge.