Generative AI and Avoiding School Boy/Girl Mistakes (October 2023)
I am hugely excited about the potential of Generative AI in legal. We have worked on some really fascinating projects which could be game changing with more in the pipeline. I am however trying to approach this area with some realism. From some comments I have heard however some run the risk of making school boy/girl errors which I would prefer we avoid as a profession.
Historically, unlike other professions the Legal Profession hasn’t made a huge mistake or been the subject of a scandal. Remember Enron in accountancy, the Year 2000 bug in technology and the MMR jab in the medical profession. Legal has thankfully not had a similar moment. We need to ensure this continues.
Set out below are a few areas we need to think about:
Quality – It is obviously vitally important that the documents that lawyers produce are accurate and they don’t have errors. For this reason lawyers are often precedent focused (i.e. they use precedents that have been tested and which have evolved over the years and/or document automation systems which are configured to avoid errors.)
Here enters Generate AI. Whilst it is wonderful to see documents being produced at the press of a button there are things we need to avoid. I am hearing some lawyers getting very excited about the benefits that Copilot will bring (e.g. the ability to draft something based on previous work products or other information in your Microsoft Tenant and also Azure AI where some lawyers are keen to populate it with huge swathes of documentation.)
All of this however ignores a basic problem in that if for example you are a non-contentious lawyer you produce first drafts which are geared up to particular circumstances whether it be bargaining position or the nuances of the deal. You then negotiate these through several drafts and the final version is a negotiated document. If you want to point Generative AI at these documents to answer questions about a particular deal then there is value in this albeit you need to choose the drafts you are focusing on carefully. If however you want to point Generative AI at these documents to generate new documents it is likely you will come unstuck.
Year after year there have been cases of negligence where lawyers used a version of a document they have used before as opposed to going back to the original precedents with the result that things are missed. In short despite the temptation to “go large” the more specific the content that Generative AI is based on the better in these circumstances (unlike the training of large language models which need huge datasets). I also suspect those firms who have not been good at opening files and particular matters may suffer due to this as their work type data may be corrupted. Data hygiene will become more important.
Prompts – Many people are increasingly realising the benefit of focused prompts but when for example Copilot is available to everyone, will they have the rigour to apply these? As per the above it is important to focus on very specific documents or document sets and failing all else precedents - again when drafting a document there are many other concerns such as who are you acting for, the strength of bargaining position, the sector in which you are acting, the extent to which you want something to be in plain English or technical – the list goes on. I am not saying it is not possible to get this right. What I am saying is it will take effort, thought and training.
The current information on Azure AI indicates there will be a lot of flexibility about the interface, APIs and content but within Copilot there will be much less. Prompt interfaces to deal with different scenarios will be hugely useful. There is also a lot of dialogue going on about how firms can integrate their document and email management systems. Again this is work in progress but we should be careful what we ask for. If we have questions relating to information from historical matters then this may be useful. If the requests are wider about generating documents for new matters then the outputs may be increasingly unreliable and it might take a more experienced lawyer to recognise this.
Risk – We need to remember that not all legal work has an equal risk profile. Yes, many firms want to develop products in advanced areas of law but there is a huge difference in the risk profile of different matters and the diligence that needs to be applied. A simple way of approaching this (which some are not doing) is to use a simple risk/benefit matrix. To state the obvious if something is low risk and the benefits are high you should be looking at it. If it is low risk and no benefit it is probably not worth your time etc. What I am saying is this is a process that every practice area needs to go through but sadly some are not. Gen AI shows huge promise but we can’t just rely upon it for everything yet. Also, as a working assumption people should remember the more valuable the deal the more likely the documents will be scrutinized retrospectively.
Time Savings – There is no doubt that Generative AI can save huge amounts of time. Again, we need to have a reality check though. If a document is based on a precedent you will know the underlying structures and mechanics. Yes, you are manually amending it but you know that the document will be solid and performant. The same applies to document assembly. A properly coded and tested document can be relied upon if you are happy with the answers to questions that have been given as there is an architecture and the underlying precedent will be solid. A Generative AI document, however, needs to be properly reviewed (or as a minimum verified by other AI). There still may be time savings but as any lawyer will know, often it is quicker to produce a document than to review and mark-up one sent to you. What I am saying is every situation will be different and we need to look at efficiency on a total time and effort basis. Just because you have generated a first draft quickly that does not mean the total time to deliver that document with the requisite quality and supervision will be less.
Risk and Liability – We seem to have entered a slightly parallel universe in that when you look at many SaaS providers, publishers and Gen AI providers, there are often very low limits on liability for outputs. For lawyers the position is very different. They are held liable for getting things wrong and accordingly have very expensive indemnity insurance. In our experience the insurers have been very good about Generative AI but have still made it clear that matters need to be closely supervised and there needs to be evidence of supervision on files. This accentuates the points made above that yes Generative AI might produce a document more quickly but the total cost of production of a document (together with supervision and verification of the same) may be higher. Some have commented how lawyers may start to do work at different price points with different liability limits depending on how it is produced. I suspect this is likely as the profession can’t be left as the only people who are liable for anything provided by Generative AI.
Changing Market – An observation on the past is that people often look at efficiency in their business model on an “as is” basis (i.e. they imagine their business will operate in a similar way to how it has in the past or perhaps an extension of evolution of the same). The change we could be going through here could be quite radical though (i.e. see The Future of Law Post Generative AI (October 2023) — Hyperscale Group Limited). If this is the case the nature of how we work may be very different (i.e. if every client and SME business can generate documents and/or has access to Copilot with the Thomson Reuters plug-in, why would they ask lawyers to generate documents?). They may however recognise that lawyers have a mastery of truth and accuracy and may ask them to verify them which will give rise to very different work with a different risk profile which has to be delivered with very different tools. In short law firms and in-house teams may look very different with different work profiles and working methods. Again, the above goes to the point that Generative AI may not deliver the efficiency savings for law firms that they first imagine.
Value - Client perceptions of value will also alter. A number of you will have heard my example in the past where on its face email could deliver massive efficiencies for a business when looked at in isolation. When however, looking at this from the perspective of everyone in the world having email, it became more inefficient due to the exponential rise of the amount of content and communication. This applies to Generative AI too. We also need to recognise that by everyone having these tools the dynamic will be very different – as a minimum content will be created more quickly and organisations will have to deal with this (either with people or with new AI tools or generative AI ).
Conclusion
To conclude I am firmly convinced we are going through an era of huge change and Generative AI will have a material impact. It can bring huge benefits but the profession does however need to be careful. We need to make sure we don’t rush into this blindly and don’t make schoolboy/girl errors. There are lots of lessons from the past we need to take on board. We may only have a small number of chances to shape what our futures look like.
Related articles:
The Future of Law Post Generative AI (October 2023) — Hyperscale Group Limited
Hyperscale Group are an Independent Digital, Innovation and Operational Advisory and Implementation Business with over 25 years plus deep market experience. We work for In-House Legal Teams and Professional Services Firms all around the world and support them developing and implementing their strategies. We help our clients to make the right things happen.
Our Experience: Click here - Our Services: Click here - Client Testimonials: Click here
How We Work: Click here
For more information, please contact Dereksouthall@hyperscalegroup.com
