Robert Ambrogi Blog
The AI legal research startup Descrybe today launched a “legal reasoning” product, DescrybeLM, that it says outperforms leading general-purpose AI models on a standardized legal reasoning benchmark — and it is publishing the methodology and scoring data to invite scrutiny.
The company also launched an all-new website that features the new product while also retaining all the functionality of its prior “Legal Research Toolkit,” which includes tools for conducting legal research by concept, keyword, case name, citation, and legal issue.
As the company says, DescrybeLM and the Legal Research Toolkit are “built to work together,” with the latter used to find the relevant law that bears on a question and the former then enabling users to reason through it against the specific facts of the matter at hand.
Benchmarking Against General AI
The company tested its new system against ChatGPT 5.2, Claude Opus 4.5 and Gemini 3 Pro on 200 questions from the National Conference of Bar Examiners MBE Complete Practice Exam. DescrybeLM answered all 200 correctly. The general-purpose models each missed between 13 and 23 questions, achieving accuracy rates ranging from 88.5% to 93.5%.
Rubric-scored reasoning quality — a separate measure evaluating whether systems correctly identified governing legal rules and applied them to the facts — followed a similar pattern. DescrybeLM scored 99.70% on that dimension. ChatGPT 5.2 scored 93.41%, Gemini 3 Pro scored 91.45%, and Claude Opus 4.5 scored 89.03%.
A central fcous of the study was not just whether AI systems get legal questions wrong, but rather how they get them wrong. Among the 52 incorrect outputs produced by the three general-purpose models, 49 were flagged as “confidently wrong” — assertive, fluent, well-structured responses that gave no signal of uncertainty. The dominant failure patterns were applying the wrong legal standard to the facts, or applying the correct standard incorrectly.
“When these systems were wrong, they were confidently wrong,” the benchmarking study said. “Among the 52 total incorrect outputs, the dominant failure patterns applied the wrong legal standard or misapplied the correct one, while presenting the analysis in fluent, well-structured prose. These are precisely the errors that impose the highest verification burden on practitioners.”
The study also found that cross-checking between general-purpose models is an unreliable safeguard. Across the 200 questions, 40 were missed by at least one of the three models, but only one question was missed by all three. Because errors were largely non-overlapping and unpredictable, model disagreement does n0t reliably identify which output is correct, it only signals that verification is needed.
The scoring log found that two general-purpose models — Claude Opus 4.5 and Gemini 3 Pro — were flagged for overconfidence on correct outputs as well as incorrect ones. Claude Opus 4.5 received three overconfidence flags total, one on a wrong answer and two on correct ones. Gemini 3 Pro received one overconfidence flag on a correct answer. ChatGPT 5.2 and DescrybeLM received zero overconfidence flags across all 200 outputs.
The study interprets this as a model-level stylistic tendency, not simply a byproduct of being wrong. A system that applies the same assertive tone regardless of whether its answer is correct, the white paper argues, gives legal practitioners less reliable signal from output confidence alone.
“We had a thesis that purpose-built legal AI produces meaningfully different results for legal reasoning tasks,” said Kara Peterson, Descrybe’s cofounder and CEO. “Legal professionals deserve to make tool decisions based on real evidence, which can be hard to find. So, we tested ourselves.”
Peterson said she understands that vendor-produced benchmarks invite scrutiny. “That’s why we published our methodology and invite anyone to replicate it,” she said.
What’s Behind DescrybeLM
Descrybe describes the focus of this study, its new DescrybeLM, as a legal reasoning engine and drafting workspace. It enables users to receive authority-grounded analysis to complex legal questions and then to refine that analysis through clarifying follow-ups.
(Descrybe invited me to test the new product in advance of today’s launch. Because of my own tight schedule, I was unable to do so, but I still plan to at a later point and will report back when I do.)
It is built, the company says, on a curated primary law corpus of more than 100 million structured records, processed at a scale requiring more than 100 billion tokens of preparation. The system is designed to produce verification-friendly outputs that include clear rule statements, application to key facts, and structured reasoning.
You can see it demonstrated in this video:
“Most AI tools are built for general use and adapted for law,” said Richard DiBona, cofounder and CTO. “DescrybeLM was built differently: from the foundation up, specifically for legal reasoning, on more than 100 million structured records individually cleaned and organized for that purpose.”
In the study, the company chose to benchmark only against general-purpose models rather than other AI platforms specifically built for legal research. I asked Peterson why that was.
She explained that the company’s central question was what distinguishes foundation models from purpose-built tools, and that the best way to test that thesis was to put themselves directly in comparison against those models. Peterson emphasized that she strongly encourages other legal AI vendors to run the same benchmark using the same methodology.
Caveats the Company Itself Raises
Descrybe is transparent about the possible limitations of its own study, which it spells out in the report.
Most notably, the company says that it cannot rule out that some or all of the 200 questions appeared in the training data of the evaluated systems, including its own DescrybeLM. The MBE Complete Practice Exam is a commercially available product. That caveat applies equally to all systems tested, but a perfect score on a published question set will invite more scrutiny than a score of, say, 93%.
The study also discloses that fine-tuning performed on DescrybeLM before the benchmark was conducted on a separate NCBE product, not the question set used in the evaluation, and that the NCBE answer key was not provided to the system during testing.
Among other limitations: the benchmark covers only multiple-choice legal reasoning and does not test drafting, jurisdiction-specific research, citation accuracy or other real-world legal workflows. The evaluation was conducted entirely by the team that built DescrybeLM. Scoring was executed by an AI judge model — specifically, GPT-5.2 extra high reasoning, which is in the same model family as one of the evaluated systems. And each question was run only once per system, meaning the results do not come with confidence intervals or variance data.
On the rubric design, the company acknowledges the possibility of unconscious bias favoring DescrybeLM’s output style, but says it mitigated that risk through three measures: the rubric was authored by a human subject-matter expert and pre-committed before scoring began; all outputs were anonymized before the judge model applied the rubric; and the same rubric, judge prompt, and settings were applied identically across all 800 outputs.
DescrybeLM’s perfect accuracy score did not imply perfect reasoning. Twenty-seven of its outputs received rubric scores below 100, mostly for incomplete distractor discussion or alternative doctrinal framing — that is, reaching the correct answer via a different but defensible legal standard than the one emphasized by the reference answer. No DescrybeLM output was flagged for wrong rule, misapplied rule, misread key fact, or internal contradiction.
Inviting Replication
As noted, the company has published its full methodology, scoring rubric, standardized prompt and a per-output scoring log covering all 800 model outputs across the four systems. The NCBE question set is commercially available, and Descrybe says any researcher or vendor can purchase it and independently replicate the benchmark.
Because of model non-determinism and ongoing provider updates, exact numerical replication is unlikely, the company says, but directional replication, including rank ordering and approximate accuracy ranges, is the expected standard.
“I’ve worked in legal technology for a long time,” said Ken Friedman, founder of the legal tech practice group at law firm L&F Brown and a strategic advisor to the company. “It’s rare to see something that genuinely stops you in your tracks. When I saw DescrybeLM answer all 200 multistate bar exam questions correctly while ChatGPT, Claude and Gemini each missed double digits, that’s exactly what happened.”
Here is the full white paper: Beyond Confidently Wrong: How Purpose-Built AI Mitigates Legal Reasoning’s Hidden Risk.