AT
Ava Thompson
1 month ago
claude wrote a rubric that graded citations that don't exist
I asked Claude (Sonnet, $20/mo Pro) to draft a grading rubric for an 8th grade research essay. Looked sharp. Five criteria, point weights, even sample comments.
Then I spot-checked one "exemplar" citation against the school library catalog. Author didn't exist. Journal issue was from 2027. The rubric still awarded full marks for "accurate sourcing."
I almost emailed it to three teachers on Friday. Now every AI rubric goes through a fake-citation hunt first. Anyone else catching invented homework scaffolding?
6 comments
Join the discussion
Log in to comment.
same energy as meeting notes that invent action items. if the brief looks polished and the sources are wrong, trust the sources. i keep a graveyard of cancelled $20/mo tools for this exact reason.
measure it. on a 20-essay sample i got 6 invented DOIs and 2 authors with plausible names who publish nowhere. coverage without a citation check is worse than a sparse rubric. false-positive rate beats vibes every time.
same numbers energy on the growth side. I ran a 15-slide client deck through Claude for "source polish" and it invented two McKinsey quotes with real-looking years.
cancelled the $20/mo seat that night. fake sources in a paid deck cost more than the subscription. citation check or it doesn't leave the draft folder.
I almost shipped the same thing last month for a Lisbon tutoring pilot. Sonnet drafted a lovely "source quality" criterion, then pointed kids at a journal that doesn't exist until 2028.
Now every AI rubric goes through a DOI ping first. Looks polished != is safe. Did you catch it before the teachers emailed, or after?
I don't grade essays but I ship decks. Sonnet put a "Forrester 2026" quote in a client one-pager last Tuesday. Forrester hasn't published that report. CAC slide looked perfect.
Cancelled the Pro seat same afternoon. If a rubric can award full marks for a DOI that 404s, it's not a rubric — it's a confidence costume. DOI ping before teachers see it is the bare minimum.
same failure mode, different department. i had claude draft a "source quality" score for a partner one-pager and it cited a HubSpot case study from q3 2027. the link 404'd. cac looked great anyway.
now every ai-touched deck gets a link crawl before it leaves drafts. teachers emailing about fake journals is bad; a client asking why forrester never wrote that report is worse. did the doi ping catch it or did a teacher?