ChatGPT-4, a member of the Large Language Model (LLM) family, has passed the Uniform Bar Exam, with impressive growth over ChatGPT version 3.5’s non-passing score. In the wake of this accomplishment and OpenAI partnering with various tech companies to beta-test new tools at select firms, the news is a continuous onslaught of dire predictions about how soon paralegals and lawyers will be declared obsolete. Okay, we’re exaggerating slightly, but then again, some of the headlines are also hyperbolically inclined. To be fair, we’re thrilled that ChatGPT helped save a dog’s life (https://www.yahoo.com/lifestyle/owner-claims-chatgpt-saved-dog-124312857.html). On the other hand, as of this date, we don’t think there’s an LLM that’s lawyer material, yet…
There are already LLM-enabled tools that can draft pleadings, contracts, memos, and outlines as part of case or topic-specific assignments. These tools also perform document review and analysis of potentially hundreds to thousands of pages in a remarkably short time. Capable of summarizing pleadings or opinions, conducting research, outlining contract terms, and creating contract workflows, corporations and legal services firms are seeking ways to integrate these time and potentially money-saving tools. What firm wouldn’t want to free up attorneys and paralegals to move on to new and more rewarding tasks faster? For centuries, it has made sense to employ tools that make everyday tasks less resource-consuming.
The jury, however, is still out on the quality of the legal product being produced. While LLM-enabled legal tools can accomplish tasks quickly, there is still the problem of LLM hallucinations – false answers presented with authority. Additionally, Open AI admits, as far as ChatGPT is concerned, that as the complexity of the task and the length of the output increases, reliability decreases. In a way, it would appear that LLM-enabled legal tools must grapple with the same issue that attorneys and paralegals do when faced with generating long and complicated pleadings: by the end, focus wanes. To avoid costly mistakes, lawyers and paralegals will continue to need to review the work product produced by LLM-enabled tools for accuracy and completeness for a while or risk having these mistakes undermine their work.
As of now, LLMs don’t have live access to the internet as a resource. For instance, ChatGPT was trained on material produced prior to the end of its training in August 2021 (https://cdn.openai.com/papers/gpt-4-system-card.pdf). Unless and until LLMs are routinely and frequently updated, returns of outdated, superseded, and overturned information will grow as the knowledge base ages. Granted, an LLM focused on the legal industry (laws/regulations, case law, etc.) could be retrained or even continuously trained, provided the developer is willing to make that investment. If your firm is considering an LLM-enabled legal tool, this might be an aspect to consider when evaluating options.
Finally, there is a facet of being human that LLMs have yet to develop: discernment. With experience, paralegals and lawyers gain intuition and skills that allow them to identify relevancy and make judgments regarding what’s the most important factor, or which of the arguments available is the best for this or that situation. Despite all the assistance LLM-enabled tools can provide (and that’s potentially a great deal), until they can make those subtle distinctions, they’re not going to be lawyer material, a conclusion with which many analysts agree.
We see tremendous potential for LLM-enabled tools to advance the legal industry, especially faster and easier drafting, enabling firms to offer lower-cost services to those currently underserved. According to the LSC, “Low-income Americans do not get any or enough legal help for 92% of their substantial civil legal problems,” (https://justicegap.lsc.gov/resource/executive-summary/). If there’s an opportunity to shrink that number, that’s one more point the legal industry should consider in embracing the new technology. And we think that’s a great reason, among many, to continue to educate and mentor young lawyers.
