I have made no secret of the fact that I consider litigation analytics to be one of the most important technologies to have gained traction in recent years. Writing about analytics a year ago on Above the Law, I titled the post, This Tech Can Turn the Tables in Litigation. In my year-end summary of the most important legal tech developments of 2018, my top item was “analytics become essential,” noting, “We could be nearing a point where it would be malpractice for a lawyer not to use analytics.”

While all of that still holds true, and while the technology has continued to evolve and the array of products to expand in the year since then, it is important for legal professionals to understand that litigation analytics is still a developing technology, and that there are weaknesses in both the technology and the data on which they rely.

That means that these products differ not only in the kinds of analytics they offer, but also in the results they deliver.

The differences among these products were dramatically highlighted by a study conducted earlier this year by a group of law librarians who compared federal court results across seven legal analytics products. The products they compared were: Bloomberg Law, Docket Alarm Analytics Workbench (from Fastcase), Docket Navigator, Lex Machina, Lexis Context, Thomson Reuters Monitor Suite and Westlaw Edge.

Consider this seemingly simple research query: “In how many cases has Irell & Manella appeared in front of Judge Richard Andrews in the District of Delaware?” When the librarians tested that query across the seven products, they got widely different answers.

The librarians tested 16 “real world” research questions across the seven products, with similarly varying results. Although they have not yet published their findings, four of the participants — Diana Koppang, director of research & competitive intelligence at Neal, Gerber & Eisenberg; Tanya Livshits, director of research services at Irell & Manella; Kevin Miles, manager of library services at Norton Rose Fulbright; and Jeremy Sullivan, manager of competitive intelligence & analytics at DLA Piper — presented their findings during an analytics “deep dive” session in July at the annual meeting of the American Association of Libraries.

(I was a participant in that panel, as was Jean O’Grady, who wrote about the analytics study in a post at her Dewey B Strategic blog: What Do Law Firms Need to Know About Buying Litigation Analytics Products?)

The librarians limited their study to federal district courts, even though some of these platforms also provide analytics for state courts. They also limited their focus to docket analytics, meaning analytics based on federal docket data, not content analytics that focus on language and citation patterns in judges opinions, such as Precedent Analytics in Westlaw Edge and Context from LexisNexis.

One reason for the divergent results across platforms, the librarians found, was what they called “the PACER problem.” In a nutshell, the problem with PACER data is that it is replete with problems. PACER data is full of typos. The Nature of Suit codes used to label types of matters often do not reflect the true focus of a matter. Names of parties, firms and attorneys are not normalized for consistency. Attorney moves sometimes result in matters being associated with the wrong firm.

(The problems with PACER data were discussed in greater depth in a guest post here earlier this year by Josh Becker, chairman of Lex Machina and head of legal analytics at LexisNexis: Guest Post: Not All Legal Analytics Tools Are Created Equal.)

There are also major differences among the products in the search options and types of analytics they provide. For example, here are how the products compared in their ability to show analytics regarding case and motion outcomes:

LITIGATION OUTCOMES Bloomberg Law Docket Alarm Docket Navigator Lex Machina Lexis Context Westlaw Edge Monitor Suite
Outcome or Findings No Yes Yes Yes Yes Yes Yes
Motion to Dismiss Yes – Judge Level only Yes Yes Yes Yes Yes Yes
Motion for Summary Judgment Yes – Judge Level only Yes Yes Yes Yes Yes Yes
Motion for Class Certification Yes – Judge Level only Yes No No Yes Yes No
Appeal Outcomes Yes – Judge Level only Yes No No Coming 2020 Yes No
Damages No Yes Yes Yes No No No
Remedies No Yes Yes Yes No No No
Jury Verdicts No Yes Yes Yes Yes Yes No
Expert Witness Challenges No No Yes (but not by expert name; using keywords over judge ruling data) Yes (only for Product Liability cases) Yes Yes (but not by expert name; using keywords over judge ruling data) No
Are additional motion type outcomes available? No Yes Yes Yes (practice area specific) Yes Yes No

And here is how they compared in terms of types of search options they offered:

SEARCH OPTIONS Bloomberg Law Docket Alarm Docket Navigator Lex Machina Lexis Context Westlaw Edge Monitor Suite
Search by Party: Company Yes – but limited (70,000 public and 3.5 million private companies) Yes Yes Yes Coming 2019-Q4 No Yes
Search by Party: Individual No Yes Yes Yes Coming 2020 No Yes
Search by Firm Yes (only those in Bloomberg law firm database; approximately 7,000 firms) Yes Yes Yes Coming 2019-Q4 Yes Yes
Search by District Court Judge Yes (current) Yes (current and retired) Yes (current and retired) Yes (current and retired) Yes Yes (current and retired) Yes (current and retired)
Search by Bankruptcy District Court Judge No Yes No Yes No No
Search by Magistrate Judge No Yes Yes (current and retired) Yes (current and retired) Yes Yes (current and retired) Yes (current and retired)
Search by PTAB Judge No Yes Yes (current and retired) Yes (current and retired) No No No
Search by ITC Judge No Yes Yes (current and retired) Yes (current and retired) No Yes (but analytics are limited)  (current and retired) Yes (current and retired)
Search by Delaware Court of Chancery Judge No Yes (partial coverage; full coverage forthcoming) No Yes (current and retired) Yes Yes Yes (current and retired)
Search by Attorney Yes – but limited (over 100,000 attorneys at over 775 law firms) Yes Yes Yes Forthcoming 2020 Yes Yes
Search by Expert No No Forthcoming Yes (Product Liability only) Yes No No
Search by Keyword No Yes (across all content) Yes (across all content) Yes (across all content) Yes (but only after selecting a judge or expert, keyword filtering available) Yes, but only within each “tab” – not across all analytics content No

It is important to note that these charts, as well as all of the study’s findings, were current as of July 15, 2019. As I noted above, and as the study authors noted, these products are evolving and changing rapidly, so they may now have capabilities or coverage not reflected above.

Another finding of the study was that the products varied in their ease of use and functionality — and there appeared to be an inverse relationship between the two factors. Here is the spectrum of the two, as presented during the AALL program:

Ease of use from least difficult to more difficult:
Bloomberg >> Context >> Monitor Suite >> Docket Navigator >> Edge>>Lex Machina>> Docket Alarm

Functionality from least complex to most adaptable: 
Bloomberg >> Context >> Lex Machina >> Docket Navigator >> Edge >> Monitor Suite >> Docket Alarm

Diana Koppang, a principal organizer of this study, tells me that she hopes to publish the full results at some point. Also, she will be presenting an update on this research in a program scheduled for Legalweek in New York in February.

But at the program last July, the four presenters summed up their “10,000 foot overview” of the findings from their research:

  • There is no winner among the products.
  • The best platform for your organization depends on your use case and budget.
  • Analytics platforms generate better results when combined with research through other platforms (company research, docket searches).
  • “Litigation Analytics” is now too broad of term.
  • “Analytics Research” is different from “Case Law Research.”
  • Assume nothing.
  • Transparency is absolutely necessary.
  • Remember the PACER Problem.

For me, the bottom line is that often-repeated Latin phrase caveat emptor. As legal professionals, we have an obligation to understand the benefits and risks of the technology we use. Litigation analytics offer many potential benefits, but if you are not aware of the shortcomings, you may also be putting yourself or your client at risk.

Photo of Bob Ambrogi Bob Ambrogi

Bob is a lawyer, veteran legal journalist, and award-winning blogger and podcaster. In 2011, he was named to the inaugural Fastcase 50, honoring “the law’s smartest, most courageous innovators, techies, visionaries and leaders.” Earlier in his career, he was editor-in-chief of several legal publications, including The National Law Journal, and editorial director of ALM’s Litigation Services Division.