Over the past several days Rebecca Fordon has been digging into the Claude for Word add-in, and the headline finding surprised her. On document-intensive legal work — cite-checking, consistency review, Table of Authorities assembly — it seems to need less supervision than either Claude on the web or Claude Code. Four tests bear that out, with limits worth knowing.
Over the past three years, researchers have published dozens of studies examining exactly when and why AI fails at legal tasks—and the patterns are becoming clearer. The research is clear: AI hallucinations in legal work are real, measurable, and follow predictable patterns. Rebecca Fordon evaluates the data and research that documents six critical patterns lawyers must understand to make sound, actionable and effective decisions about using AI.
It is difficult to test Large-Language Models (LLMs) without back-end access to run evaluations. So to test the abilities of these products, librarians can use prompt engineering to figure out how to get desired results (controlling statutes, key cases, drafts of a memo, etc.). Some models are more successful than others at achieving specific results. However, as these models update and change, evaluations of their efficacy can change as well. Law Librarians and tech experts par excellence, Rebecca Fordon, Sean Harrington and Christine Park plan to propose a typology of legal research tasks based on existing computer and information science scholarship and draft corresponding questions using the typology, with rubrics others can use to score the tools they use.
Is better case law data fueling a legal research boom? Recently, Rebecca Fordon noticed a surge of new and innovative legal research tools. Fordon wondered what could be fueling this increase, and set off to find out more.
The pace of generative AI development (and hype) over the past year has been intense, and difficult even for us experienced librarians, masters of information that we are, to follow. Not only is there a constant stream of new products, but also new academic papers, blog posts, newsletters, and more, from people evaluating, experimenting with, and critiquing those products. With that in mind, Rebecca Fordon shares her favorites, as well as recommendations from her co-bloggers.
Hallucinations in generative AI are not a new topic. If you watch the news at all (or read the front page of the New York Times), you’ve heard of the two New York attorneys who used ChatGPT to create fake cases entire cases and then submitted them to the court. After that case, which resulted in a media frenzy and (somewhat mild) court sanctions, many attorneys are wary of using generative AI for legal research. But vendors are working to limit hallucinations and increase trust. And some legal tasks are less affected by hallucinations. Law Librarian and attorney Rebecca Fordon guides us to an understanding of how and why hallucinations occur and how we can effectively evaluate new products and identify lower-risk uses.
What is RSS and how do federal courts use it? Rebecca Fordon informs us that courts vary in the types of documents they provide via RSS feeds – only about 70% of bankruptcy courts and 50% of district courts provide full feeds. The effort urging courts to fully enable RSS feeds has many advocates and would have a significant positive impact for legal researchers in all sectors.