Over the past few years’ advancements in Artificial Intelligence have taken the world by storm, the impact of which has been compared to the advent of the internet or even the discovery of electricity. The ROSS Intelligence team has been part of this wave, having built the world’s first artificially intelligent legal system, created to enhance the capabilities of lawyers, thereby allowing them to do more than humanly possible. Notably, just last week CB Insights recognized ROSS as one of the world’s top 100 A.I. companies.

Alongside this rapid advancement and excitement about A.I. there has been a lack of evidence-based assessments of how new A.I. tools will impact business operations and, indeed, day-to-day life in a concrete way. This evidence deficit has had a particularly acute impact on the legal field, where lingering suspicion and cultural reluctance is paramount, specifically with respect to disruptive innovation and advancements in legal technology — hence this post’s title, “A.I. is hype! What can it really do in law?”

Given these realities, the ROSS Intelligence team decided to commission a third-party, evidence-based study of the impact the ROSS A.I. tool can have on the successful practice and business operations of legal organizations. We engaged Blue Hill Research, a leader in verifying technology-driven business success, whose reports are often cited by other legal technology companies due to their extensive experience analyzing the legal research industry. Blue Hill prepared a third-party report comparing the respective impact of traditional legal research tools created by Westlaw and LexisNexis, such as Boolean search and Natural Language search, with the ROSS A.I. tool.

I decided to write this blog post on the ROSS LegalTech Corner to help provide a quick overview of the Blue Hill assessment, highlighting key takeaways so that folks have a digestible preview of the full report, which is a must read and can be downloaded on the ROSS website.

I focused on the following three key takeaways:

1. The ROSS A.I. tool provides significant, additive contributions to the effectiveness of legal researchers;
2. The report revealed significantly stronger indicators of user satisfaction with and confidence in the ROSS A.I. tool when compared to those users restricted to using only Westlaw and LexisNexis Boolean or Natural Language search tools;
3. With adoption of the ROSS A.I. tool, a positive return on investment (ROI) can be obtained with as low as a 10% conversion rate of unbillable to billable time

Blue Hill’s research indicated “clear advantages resulting from the use of ROSS to supplement traditional electronic legal research practices”, ultimately concluding that the ROSS A.I. tool can “unlock new gains in the efficient and profitable operation of legal organizations, as well as create opportunities for new revenue gain.”

1. LEGAL RESEARCH EFFECTIVENESS GAINS WITH ROSS
2. “Blue Hill finds that the ROSS tool provides significant, additive contributions to the effectiveness of legal researchers”
3. Blue Hill found that the “ROSS tool provides significant, additive contributions to the effectiveness of legal researchers.” These gains include between a “30.3% and 22.3% reduction in research time” compared to groups relying on traditional Boolean search alone or Natural Language search alone, respectively, which reductions stem “from substantial improvements in information retrieval, particularly in the ranking of research results identified by a 0.61 NDCG score.”
4. NDCG, short for normalized discounted cumulative gain, is a widely used ranking measure used to evaluate the performance of a ranking function, such as a web-based search engine where the importance of having the most relevant documents ranked at or near the top is of utmost importance to users. NDCG is represented by a number between 0 and 1.0, with 1.0 representing an ideal or perfect ranking of the entities.
5. “Blue Hill finds that the ROSS tool provides significant, additive contributions to the effectiveness of legal researchers. These gains include between a 22.3% and 30.3% reduction in research time, stemming from substantial improvements in information retrieval, particularly in the ranking of research results identified by a .61 NDCG score.”
6. For context, Blue Hill found that the 0.61 NDCG score obtained by the ROSS A.I. tool was “46.1% higher than natural language search and 127.8% higher than Boolean.” This led Blue Hill to conclude that “[f]or the user, this means that not only is the ROSS tool generating more relevant results with less noise than Boolean or natural language search, these results are highly concentrated within the first results that the user sees.”
7. These numbers reflect the true power of A.I.-assisted search engines like ROSS. By truly understanding a lawyer’s legal research question, rather than merely focusing on key words or Boolean search strings, succinct and relevant results will be provided at or near the top of a list of search results. Busy lawyers with little time on their hands can ask their legal research question just like they would ask it to a colleague down the hall and ROSS is able to answer that question quickly and with confidence. Lawyers using ROSS no longer have to spend hours and hours digging through hundreds or thousands of irrelevant search results in search of answers.
8. Moreover, being an A.I.-tool, ROSS is dynamic, not static — it learns and gets better and better over time. As the ROSS A.I. tool continues to learn, the ROSS team has every confidence that its NDCG score will continue to move closer and closer to the 1.0 ideal. This is a powerful prospect considering that after only a short period of time and as a start-up, the ROSS NDCG score already significantly exceeds those of Boolean search and Natural Language search.
9. USABILITY AND USER CONFIDENCE SIGNIFICANTLY HIGHER IN ROSS
10. “In nearly all cases, the responses indicated by participants using ROSS and another tool often exceeded those of organizations using only Boolean search or only Natural Language search by at least a full point.”
11. In addition to assessing the impact of the ROSS A.I. tool with respect to information retrieval quality, research efficiency and the ultimate impact on business value and ROI, Blue Hill evaluated user perceptions regarding their confidence in and the effectiveness of the Boolean, Natural Language and ROSS A.I. legal research tools. The users were asked to indicate their relative agreement or disagreement with a list of statements on the 5 point Likert scale, with 1 indicating strong disagreement and 5 indicating strong agreement.
12. The results revealed (1) “strong indicators of user satisfaction of the users using a ROSS-supported toolset with respect to the usability, presentation of search results, and inclusion of relevant authorities within the search results”; (2) “high levels of confidence in the ability of the tool to identify all authorities relevant to the matter” and ultimately (3) “[i]n nearly all cases, the responses indicated by participants using ROSS and another tool often exceeded those of organizations using only Boolean search or only Natural Language search by at least a full point.”
13. Because of the higher satisfaction with the ROSS A.I. tool, over the course of the study those participants in the ROSS-assisted groups “tended to increase use of and reliance on the ROSS tool as they progressed through the question sets.” This indicates that even in a short period of time, researchers gained trust in an A.I. solution, with it quickly becoming obvious to them how much an A.I. tool could assist them with their legal research. Users were able to discern that the results produced by ROSS were concise, primarily contained cases that were relevant to the legal questions asked and they were confident that the tool returned all of the cases required to give a comprehensive answer. This suggests that when dealing with A.I. solutions, like hiring a new human lawyer to a team, interaction and use brings out trust and reliance.
14. ADOPTION OF ROSS LEADS TO POSITIVE ROI AT MINIMAL INVESTMENT
15. With adoption of the ROSS A.I. tool, “a positive ROI is obtained at a 10% conversion rate” of unbillable to billable time “meaning that the investment drives a net gain to the firm with a minimal recovery of written-off hours.”
16. Legal research is an area of legal practice that clients are increasingly unwilling to pay for. Blue Hill found in their 2014 report “Calculating the Value of Legal Research” that the typical litigation associate in the United States spends an average of “743.6 hours a year completing legal research, 26% of which is written off as unbilled or unpaid by clients.” Reducing the amount of time spent on legal research and converting that time to higher-value billable work creates the opportunity for a net increase in revenue.
17. A law firm is not guaranteed to be able to achieve a total conversion of unbilled legal research time; however, even partial conversion can generate significant benefits to a firm’s bottom line. Blue Hill found that with adoption of the ROSS A.I. tool, a positive ROI is obtained at as low as a 10% conversion rate, “meaning that the investment drives a net gain to the firm with a minimal recovery of written-off hours.” Similarly, a 176.4% to 544.5% ROI becomes possible with a 25% conversion; 268.1% to 758.4% ROI with a 33.3% conversion and 452.7% to 1,188.9% ROI with a 50% conversion.

The investment impact of this A.I.-led stage of evolution is substantial and the ROSS A.I. tool represents “a clear response to the present market needs, delivering value through both cost of ownership and contributed value vectors.” Through this powerful combination, the tool is able to “demonstrate net business gains and ROI in use cases that enhance, rather than replace, traditional research strategies.”

Where exactly a law firm falls on the spectrum of conversion from unbillable to billable time will be very firm-specific, relying on a variety of factors from business-type to scope of use; however, in all cases, Blue Hill’s model “strongly suggests positive business gain is available from the investment in ROSS.”

Even those firms that continue to bill all their legal research hours will gain advantages through quicker turnaround time on client deliverables; more time spent engaging with the law, thereby speeding up the learning curve for junior associates; lower billable rates to clients for legal research, creating a competitive advantage in a market which continues to move from a sellers’ to a buyers’ market with the rise in fixed fee arrangements, RFPs and general push-back on time billed by clients.

For in-house legal teams and government agencies, adoption of the ROSS A.I. tool, with its demonstrated ease of use and efficiency advantages, allows for faster answers to legal research questions at lower cost. The main reason being that questions that used to be punted to outside counsel due to complexity can now be handled internally, taking skyrocketing law firm billables even further out of the equation.

Key Observations and Takeaways

At ROSS Intelligence we continue to push A.I. in law, and the Blue Hill benchmark report offers clear, a data-driven third-party proof point that the ROSS tool:

1. Provides significant, additive contributions to the effectiveness of legal researchers;
2. Increased lawyer satisfaction and confidence in legal research tools; and
3. Net business gains and positive ROI with low up-front investment

“A.I. is hype! What can it really do in law?”

Well, as Blue Hill’s report indicates, A.I. is not hype and A.I. can do a lot in law.

The Blue Hill report is an important step forward in addressing the cycles of hype and anxiety that have colored the initial discourse surrounding A.I. tools. Moreover, this is just the beginning. The Blue Hill report focused only on ROSS’s core search capabilities and as the ROSS team continues to add more functionalities into ROSS each and every day, the benefits to lawyers and legal organizations of adopting the ROSS A.I. tool will become more and more obvious.

‍