Robert Ambrogi Blog
The legal technology company Filevine has launched an AI-native citator and a companion brief-checking tool inside LOIS, its Legal Operating Intelligence System AI platform, that its CEO says perform as well as or better than the leading legal citators from LexisNexis and Thomson Reuters.
The citator has two core features. One checks a filed or draft brief for hallucinated citations, altered quotations, and mischaracterized authority. The other checks whether a proposition is still good law by letting a lawyer highlight a specific passage in an opinion and find out whether later courts have undercut the holding expressed in the passage.
Although Filevine had previously announced the citator in June as part of its new LOIS Legal Research offering, which provides access to millions of opinions from U.S. federal and state courts, CEO Ryan Anderson, in a briefing this week, told me the company had not heavily promoted the citator until it believed the product was fully ready for customers to use.
The company had not previously announced the anti-hallucination tool, which is being released for the first time this week.
As Filevine described the citator in its June release: “While legal verification tools rely on explicit citation relationships and apply treatment signals to entire opinions, LOIS Legal Research operates at the user-selected passage level.” This enables legal professionals to verify the exact rule they plan to rely on in their argument.
The Challenge Of Citators
I have previously written that developing a strong citator tool is the Holy Grail of a legal research platform. Among companies to have tried it in recent years were vLex and Paxton, Descrybe, and Midpage.
But it is not an easy task. In fact, in my briefing with Filevine this week, John Rizner, the product manager who led the development of the citator, pointed out something that most law librarians will know but that may be news to almost everyone else, which is that the two citators lawyers have long considered to be the gold standard routinely disagree with each other about whether a case is still good law.
He pointed to a 2018 study in the Law Library Journal by Paul Hellyer, a librarian at William & Mary Law School, who reviewed 357 citing relationships that at least one of Shepard’s, KeyCite or BCite had labeled negative.
All three citators agreed that the treatment was negative only 53 times, meaning that in 85% of the relationships in Hellyer’s sample, the three citators did not agree that a negative treatment had occurred.
Hellyer also examined whether the flags that did appear were correct. Counting relationships the citators omitted or mislabeled as positive or neutral, together with those labeled negative but incorrectly described, he found overall failure rates of 33% for Shepard’s, 38% for KeyCite and 72% for BCite.
Rizner also cited a second line of research, from Susan Nevelow Mart, the former director of the law library at the University of Colorado, whose 2013 examinations of digest and citator results found that the overlap between the Westlaw and Lexis citator systems on relevant results was small – only about a third.
As the abstract of Mart’s article put it: “Neither algorithm is doing a very good job of finding all the relevant results; the overlap between the two citator systems is not that large.”
Ahead of our briefing, Rizner ran his own test. He pulled Freedom Mortgage Corp. v. Engel on both platforms and filtered each down to negative treatment in reported opinions. Westlaw found 28 and Lexis found 13. The overlap between the two sets, by his count, was about 41%.
“If my law firm happens to have Lexis, I might be missing opinions because I don’t have Westlaw, or vice versa,” Rizner said.
What that means, Anderson said, is that even though the legacy citators have been best in class up until now, that does not mean that they do not make mistakes.
Citators As An Essential Tool
At this point, you may be thinking: “Sure, but what does this all have to do with Filevine, which is not a legal research platform?”
In my opinion, it has everything to do with Filevine and every other legal platform in which legal professionals are using AI to draft legal documents that rely on legal citations.
Thanks to concerns about AI accuracy and rampant hallucinations, the role – or maybe I should say the place – of the citator is evolving.
For most of their history, citators were legal research products – the last step before you wrapped up your research. But, as Anderson argued in our briefing, citation validation is now a required step in any legal workflow that produces a document relying on case law.
“Usually you’re taking facts, applying them to law and drafting,” he said. “That is the day-to-day shoe leather work of a lawyer.”
Filevine already held the facts, he said, but the missing pieces were a case law corpus and a way to confirm that what the corpus returns is still good law.
“And I think now there’s another step,” he said, “which is make sure nothing you’ve looked at is hallucinated.”
Earlier this year, at a meeting of roughly 20 large firms that Filevine convened, Anderson heard the same anxiety about hallucinations expressed repeatedly. One litigator said his top-10 firm now requires three separate sign-offs before any filing goes out, with the briefing associate required to attest that the associate had checked every citation and every holding.
Filevine’s Answer to This Need
So what is Filevine’s answer to this need? In our briefing this week, Anderson and Rizner demonstrated how the brief auditor and citator work.
For the brief auditor, after It ingests a document, it extracts and analyzes every citation, and then reports four things in a side-by-side chart:
- The proposition the brief says the authority supports.
- The citation given.
- What the system found in the opinion.
- The status, which can be verified, unverified, suspect, ambiguous, mischaracterized or wrong case name.
Using briefs pulled from PACER in matters where courts had already found problems, Anderson showed me examples. In one, a brief asserted that the 11th Circuit had squarely held that “100% healed” policies violate the ADA. The tool reported that the phrase appears in the opinion only in the plaintiff’s own request to return to full duty.
In another, drawn from Knight v. Baptist Hospital of Miami, the brief cited a passage for the proposition that disparate discipline of similarly situated employees is evidence of pretext. The tool flagged that the quoted sentence does not appear in the opinion and that the court’s reasoning runs the other way, having found the comparator not similarly situated.
“It’s really quite remarkable how subtle these hallucinations can be even when the case itself is not hallucinated,” he said. He noted that fabricated support often turns up in a dissent or in a recitation of case history rather than in the holding.
A Different Kind Of Citator
The citator works differently from a traditional one. Rather than starting from a case and examining its citation graph, the user highlights the specific language at issue.
Filevine then runs two passes – a traversal of the citation graph and a semantic similarity search – across its corpus for opinions using comparable reasoning. Opinions that come up as likely candidates are handed to LLMs, which distill the conflicts.
The output is a report keyed to the highlighted passage, with a severity rating and a written treatment for each authority.
In the results, a second tab generates what Rizner described as a synthetic treatise – a narrative discussion of the doctrinal tension, structured like a secondary source but built around the specific issue the user selected.
Because the semantic pass does not depend on one court citing another, Rizner said, it surfaces sub silentio conflicts, such as, for example, a 1985 decision and a 2005 decision that squarely contradict each other but never cite each other. This would be invisible to any citator built on a citation graph alone.
Evaluating the Results
Filevine gave me access to an as-yet unpublished paper evaluating its anti-hallucination tool.
According to the paper, Filevine’s evaluation team started with a database of 1,314 actual matters in which a judge had identified potentially hallucinated citations tied to admitted or suspected use of gen AI. It narrowed that pool to federal filings and then to filings submitted by attorneys rather than pro se litigants. That left 103 cases, of which 68 had records available for full analysis.
For each of the 68 matters, Filevine pulled the court’s order and the offending brief, ran the brief through LOIS using an identical prompt, and compared the output against the order. LOIS worked only from its own case law corpus, blocking general web search to keep it from finding news coverage or blog abouts about the cases.
Across 2,073 citations, LOIS verified 61.8%, flagged 38.2% for attorney review, and marked 8.4% as severe, meaning they were mischaracterized holdings or citations pointing to the wrong or a non-existent case. Every document in the sample produced at least one flag. Among the highest-citation filings, the cleanest still had roughly 9% of its citations flagged, while the worst had 76%.
Specifically, LOIS was instructed to look for three categories of hallucinated cases: fabricated case law, false or misattributed quotations, and mischaracterized authority.
For fabricated case law, LOIS found them in multiple matters. The report gave the example of Renschler v. Yosh, 909 F. Supp. 2d 1197, a citation LOIS flagged because no case by that name appeared anywhere in the reporter database.
For altered quotations, it also found multiple examples. The paper points to a Rule 37 sanctions argument that quoted the 2nd Circuit case of Shcherbakovskiy v. Da Capo Al Fine, Ltd. as requiring fault “of the sanctioned party,” when the court actually wrote “of the deponent.”
Filevine said the third category, mischaracterized authority, yielded “the exercise’s most critical observation,” which was the high number of instances when the citation appeared legitimate, but it did not stand for the proposition asserted, did not contain the argument attributed to it, or contained citations that LOIS could not access to confirm the authority.
In one filing, a brief cited Watson v. State of California for the proposition that California Government Code § 845.6 imposes an affirmative duty to summon medical care, when the opinion holds the statute confers broad immunity and the Watson plaintiff lost.
In another, Nai v. National Asset Mortgage, LLC, in the Western District of Michigan, counsel quoted RESPA’s limitations provision, 12 U.S.C. § 2614, as running from the “first” occurrence of the violation. The word “first” is not in the statute, meaning its insertion converted an occurrence-based trigger into a first-occurrence trigger, which counsel used to argue two tax years of claims were time-barred.
Notably, the report says, in matters where the bad citation had already been flagged by a court, LOIS caught the same defect, and it identified authorities cited for propositions that contradicted the underlying argument.
A Few Other Notes
As I noted above, the citator bases its analysis on the text highlighted by the user. Given that, I asked Rizner whether it was possible that a lawyer who validates one holding in a long, multi-issue opinion would not necessarily learn that the case was reversed on an unrelated holding.
He confirmed that is possible, though he said the system’s severity ratings are designed to explain when a citing case engaged with the opinion but not with the issue at hand. Anderson suggested that a direct query, or highlighting the full opinion, would likely catch it, but acknowledged the company had not tested that.
As for the source of Filevine’s case law corpus, it is using the CourtListener library, supplemented by licensed material that provides reporter citations and pagination for cases where they are lacking from the CourtListener collection.
The collection does not currently include statutes or regulations, although Filevine plans to add them in the future.
Bottom Line
I have not yet directly tested Filevine’s citator and anti-hallucination tools, although I hope to do that soon.
However, I think that this will prove to be of significant value to Filevine’s customers, many of whom are law firms with significant litigation practices.
By integrating within its AI platform both a corpus of case law and the ability to check citations against that corpus, Filevine is providing customers with an important safety check against hallucinations.
Of course, as Anderson pointed out, none of that replaces the need for lawyers and legal professionals to validate their own work.
“I don’t think it completely solves the problem,” he said. “A human still has to take a look. But at least we’re going to point you in the right direction.”