EF
Elena Fujita
8 days ago
Claude invented three citations that never existed
Asked Claude (Opus) to outline a lit review on generative tools for design research. It gave me a clean structure, then three papers that sounded perfect.
None of them exist. One had a real author name glued to a fake 2024 title. Another cited a journal issue that skips that volume entirely.
I only caught it because I keep a small spreadsheet of "looks real, isn't". Anyone else gating agent research notes with a hard "no new citations unless DOI resolves" rule, or am I being paranoid?
5 comments
Join the discussion
Log in to comment.
not paranoid. happened to me on a product brief last month — agent cited a "2023 Nielsen Norman Group study on AI onboarding" that googles to nothing.
i now paste every citation into a DOI/url check before it leaves draft. annoying but cheaper than explaining a fake source to a client.
do you make the model list sources first with links, or only check after?
i do both — force a sources block with urls first, then a script hits every link before the draft can leave the folder.
still got burned once when the model invented a plausible doi that returned 404. resolver script is non-negotiable now.
the "do not invent citations" system prompt? still invents. maybe 1 in 6 research runs on opus.
Same class of bug as agents inventing Stripe refund ids. Confident tone, zero grounding.
I put a hard stop in the system prompt: "Do not invent citations. If you lack a source, say so." Still slips through maybe 1 in 5 runs on research tasks.
Spreadsheet of fakes is a good idea. Mine is just a #agent-lies channel.
Same. Soft prompt bans leak. What actually cut our fake-cite rate was wiring web_search as a hard tool gate — model can't emit a citation unless the tool returned a hit that turn.
Do you log the fakes with model + temp? I've been dumping them into LangSmith and the pattern is boringly consistent: high-confidence prose, zero grounding, usually a real venue name glued to a made-up title.
Not paranoid. I shipped a client deck last Tuesday with two "CHI 2024" papers Opus invented. One author was real; the paper wasn't. Client googled it mid-call.
I keep a Notion page called agent-lies now. Rule is simple: no citation leaves draft unless the DOI/URL resolves. Feels slow. Beats apologizing on Zoom.