On Oct. 30, McCoy Russell LLP announced that one of its tools had not just scored higher than other similar technologies on the U.S. Patent and Trademark Office (USPTO) Bar Exam, it actually outperformed the average human.
Given the easily stoked concerns of artificial intelligence (AI) eliminating jobs, wiping out industries, and portending the technological destruction of all humanity, it’s as important in this case to talk about what the firm’s Ironcrow AI LLM Sandbox does not do as much as what it can do.
“We’re always looking for tools, what can we use to make our work better,” said firm partner and cofounder John Russell. “… We started playing around and figured out the tools are good at wide-ranging tasks, but not very good at specialized patent work.”
With a staff of about 85—most of whom are technical employees working alongside about 10 attorneys—the self-described boutique IP patent firm began building its own AI tools internally in 2018, Russell said. One of those tools grew into the LLM Sandbox which was trained, in part, on already submitted patent applications in order to evaluate new applications for problem areas that might result in a rejection from the USPTO. In that regard, said Russell, the tool works beautifully at helping a patent attorney be more efficient and effective.
So, the actual purpose of the LLM Sandbox is to play a supporting role for patent attorneys, not replace them. But the team also wanted to put their AI to the literal test.
“We got to the point that we’d put so much into this, let’s see what it does,” Russell said. “Can it pass the patent bar?”
According to McCoy Russell, the LLM Sandbox when asked a test set of patent bar exam questions “achieved scores ranging from 78% to 80%, surpassing the average human performance of 68%.” Further, the firm claims, it’s the first time such an AI passed the specialized patent bar exam.
What follows is an interview with Russell about what this achievement means, what legal professionals should take away from it, and what the LLM Sandbox does and does not do.
What significance, if any, is there to Ironcrow AI’s LLM Sandbox passing of the patent bar exam? Is it simply an impressive achievement, or indicative of AI’s future role in the law?
Part of our firm culture includes embracing innovation and applying our expertise to generate the highest quality patent portfolio for our clients. We have been developing AI tools to aid patent professionals for over 5 years, and the pace of development has consistently increased. Our Ironcrow Sandbox was developed to provide wide-ranging support to improve quality and efficiency in patent portfolio development, and the fact that it is also able to score a passing grade (higher than the average human test taker) in such a specialized legal field speaks to the power of the tool in that space. Just as word processors changed the legal office, new technology will continue to shape the practice of law.
What is this tool good at versus what types of tasks is it not intended to perform?
Our tools aim to support patent professionals, not replace them. The tool can help patent professionals more quickly and efficiently understand a particular prior art document, or what happened in a long patent prosecution history, or even point out potential amendment approaches.
However, these suggestions still require a trained patent professional. Just as AI tools are helping doctors diagnose images or processes patient medical records more quickly in pointing out additional actions, the doctor is still required to assess the situation and carry out the practice of medicine.
Even the best AI systems lack an intuitive understanding of how the world works, and while our system is able to generate helpful feedback and perform some comparative analysis, an attorney is still needed to analyze the facts and develop a legal strategy to support the client’s goals.
Should patent lawyers and other attorneys be nervous about their job security?
All attorneys have to establish a value proposition for their clients. Smaller, innovative firms fill an important gap left open by larger, slower moving firms by pushing the industry. All practitioners should strive to utilize the best tools to support their clients. Whether the rate of development of these tools will continue at this pace is yet to be seen, but fundamentally AI tools are here and unlikely to go by the wayside.
What ethical considerations were most discussed when developing this tool?
As legal professionals it is our duty to keep all client confidential information confidential and secure. That obligation lies at the forefront of ethical considerations in development of Ironcrow AI’s tools, especially the Sandbox with its interrogate feature. For example, in addition to grounding the responses of the AI system in factual sources of information pertaining to U.S. patent prosecution, we also ensured that any portions of those sources relied upon in generating the response are provided to the users, enabling them to easily verify the response, or as a potential jumping off point for further research.
