Robert Ambrogi Blog
Vals AI has put out an open call for legal technology vendors to participate in its legal AI benchmarking studies, including its upcoming evaluation of U.S. legal research tools.
Readers may recall that, in February, Vals AI issued the first-of-its-kind benchmark study of legal AI tools. The Vals Legal AI Report (VLAIR) was the first systematic attempt to independently benchmark legal AI tools against a lawyer control group, using real-world tasks derived from Am Law 100 firms.
That report did not include legal research products, but at the time, Vals said it would be issuing a subsequent report focused on legal research benchmarking.
At the time the report came out, some vendors said they had not been aware of the study and had not been given an opportunity to participate.
In response, Vals is now reaching out to encourage the broadest possible participation by companies with generative AI legal research tools, as well as in future legal AI benchmarking studies.
“We made the decision to separate out legal research from the rest of the study because we wanted to ensure we had representative vendor participation and a sufficiently diverse set of questions,” Tara Waters, project lead for the study, told me in an email. “We are collecting 200 Q&A from the law firms. As before, we will have a human baseline.”
How To Register Your Interest
Companies that want to have their product evaluated — whether in the legal research study or future studies — should complete this Vendor Interest Form.
Vals is also inviting law firms, solo practitioners, and in-house lawyers to join its benchmarking consortium. Law firms interested in participating should fill out this Law Firm Interest Form.
Waters said that the legal research benchmarking will be different from the previous VLAIR tasks in that the responses from both the lawyer group and the AI products will be assessed by human graders.
“We have a mix of law librarians and independent lawyer reviewers,” she said. “The assessment will take into account accuracy, authoritativeness and appropriateness of the responses, with weighting applied.”
Waters is also one of the principles behind the AI Adoption Index, which, as I recently reported here, aims to develop an accurate, holistic and transparent index of AI adoption among law firms, in-house legal departments, and individual legal professionals.
The Vals legal AI study was developed in partnership with Legaltech Hub and a consortium of law firms including Reed Smith, Fisher Phillips, McDermott Will & Emery, and Ogletree Deakins.