JL
Jordan Lee
5 days ago
Jev said approve_refund at 0.93 and I almost wired $400
Jev
System One model for typed decisions with calibrated confidence
Tried Jev this week for triage on support tickets that need a typed decision (refund / escalate / ignore).
It returned approve_refund with confidence 0.93 on a ticket where the customer already got a store credit. The confidence bar looked so clean I almost hit send on the Stripe refund before I checked the ledger.
Is anyone else wiring this into a money path, or keeping it draft-only until they ship a calibration set?
5 comments
Join the discussion
Log in to comment.
0.93 with no holdout number attached is a vibe, not a score. Ask them for ECE or Brier on your ticket distribution, not the demo set.
We kept Jev in draft mode for anything that touches Stripe. Escalation suggestions are useful; money actions need a human click.
yeah the pretty bar is the actual bug. we had a green 0.9 pill in the stripe refund modal and people stopped reading the ledger line under it.
killed the confidence UI, left a plain "suggested: approve_refund" text. click-through on money actions dropped and that was the point.
yeah i wired it for auto-escalate only and still almost emailed a customer a "we're issuing your refund" draft that was wrong.
$20/mo is fine until the confidence bar makes you skip the ledger check. draft-only until you have a real calibration set imo.
same near-miss last Friday. agent drafted a $180 refund on a ticket that already had a partial credit. Zapier almost fired the Stripe call before I caught it in the review queue.
draft-only forever for money. escalate suggestions are fine; anything that moves cash stays behind a human button.
Curious what your buyer demos looked like with that 0.93 bar. In Intercom trials I've watched people treat the meter like a green light and skip the ledger check every time.
Did Jev expose any calibration on your ticket mix, or was it the default demo set? Draft-only feels right until that gap is boringly obvious.