Anthropic is under scrutiny after its Claude AI chatbot was caught generating a false citation in a high-profile legal battle with music publishers. In a filing submitted Thursday in Northern California, the company admitted that one of its own legal representatives unknowingly included a hallucinated reference created by Claude in court documents.
The error came to light after opposing counsel, representing Universal Music Group and other publishers, raised concerns over questionable references cited by Anthropic’s expert witness, Olivia Chen — who also happens to be a company employee. The judge, Susan van Keulen, then ordered Anthropic to explain the discrepancy.
In response, Anthropic admitted that Claude hallucinated a citation featuring “an inaccurate title and inaccurate authors.” The company said it failed to catch the mistake during its manual review process and acknowledged that several other citation errors stemmed from the chatbot’s responses. Anthropic apologized in the filing, emphasizing that it was “an honest citation mistake and not a fabrication of authority.”
The AI Legal Citation Problem Isn’t Going Away
This isn’t the first — and likely not the last — time a generative AI tool has caused problems in court. Judges across the globe are increasingly confronting AI hallucinations in legal cases, and the implications are growing.
Just this week, a California judge criticized two law firms for submitting AI-generated legal research riddled with errors. Earlier this year, an Australian lawyer was caught using ChatGPT for court filings, only to discover the tool had inserted nonexistent citations. These incidents are raising serious questions about the use of generative AI in legal practice.
Yet, despite these high-profile missteps, startups aiming to automate legal workflows are booming. Harvey, one of the most prominent names in AI legal tech, is reportedly in talks to raise over $250 million at a valuation of $5 billion. Its tools promise to help law firms streamline research and document drafting — but clearly, quality control is still a critical issue.
The legal industry is at a crossroads. While generative AI has the potential to transform law practice, incidents like Anthropic’s highlight the risks of over-reliance. The big question now: Will courts soon require full transparency when lawyers use AI?
