AI-Generated Errors Proliferate in Academic and Legal Fields

Artificial intelligence tools are increasingly introducing fabricated information into academic papers and legal documents, raising concerns about the integrity of professional work.

Maxim Topaz, an associate professor at Columbia University's School of Nursing, discovered that an AI tool he used had inserted a fabricated source into his research paper. "I felt deeply embarrassed," Topaz said. "I'm an AI researcher. I know about hallucinations. If this is happening to me, an AI expert, what happens to other people?"

In a study published in The Lancet, Topaz and his colleagues audited nearly 2.5 million biomedical papers and 97 million citations indexed on PubMed Central. They found over 4,000 fabricated references across nearly 3,000 papers. While not all were AI-generated, the incidence of fake sourcing increased significantly in 2024, coinciding with the widespread adoption of AI tools in research.

Over the past three years, the rate of fabricated references in biomedical literature has grown more than 12-fold. In 2023, one in 2,828 papers contained at least one fake reference; by last year, this rate had risen to one in 458. In the first seven weeks of 2026, one in 277 papers had at least one non-existent reference. "I'm thinking this is just the tip of the iceberg," Topaz said.

AI hallucinations occur when models prioritize word patterns over accuracy, often producing plausible but false information. In fields like medicine, where clinical trials and guidelines build upon previous studies, such errors can undermine the scientific process. "This is the evidence chain, that's how we care for and treat people. If you put the fictional study at the bottom of the stack, the whole structure inherits it," Topaz explained.

The issue extends beyond academia. Author and filmmaker Steven Rosenbaum's recent book, "The Future of Truth: How AI Reshapes Reality," contained numerous misattributed or entirely invented quotes, apparently generated by AI tools he had disclosed using. Rosenbaum recognized the errors, calling the episode "a warning about the risks of AI-assisted research and verification."

AI tools are now prevalent in various professions. Surveys suggest more than half of legal professionals use AI to draft briefs and memos, and over 80% of physicians use AI to summarize research and prepare clinical documentation. However, even experts are susceptible to AI-induced errors. Legal analyst Damien Charlotin cataloged 1,459 legal decisions citing AI-generated inaccurate content, noting that such cases have increased from two or three times a month to around five a day.

The publishing industry may not be prepared to handle the surge of fake references. Verification methods differ between journals, and while some use software to check references and scan for AI-generated content, enforcement varies. Topaz's analysis found that 98.4% of studies with fake references had not been retracted by publishers at the time of his audit.

Topaz emphasized that AI itself is not necessarily the villain. "The problem is unverified AI output entering the permanent record," he said. "The fix is not to stop using the tools, it's to build verification into the workflow." He warned that the longer verification measures are delayed, the harder it becomes to clean up the inaccuracies.

As AI continues to permeate professional fields, the importance of rigorous verification processes becomes increasingly evident to maintain the integrity of academic and legal work.