Capabilities and Limitations of AI

Understanding the Capabilities and Limitations of AI: A Personal Reflection from Experience

Artificial Intelligence (AI) has been a transformative force across various industries, offering significant improvements in efficiency, accuracy, and innovation. However, its integration into daily workflows comes with both notable advantages and inherent limitations. In this post, I reflect on my personal experiences with AI, exploring both its beneficial uses and the scenarios where it falls short.

​

The Why?

First of all, lets quickly look into why its great at some tasks and can be useless at others. Generative AI, which produces text based on patterns from its training data, faces  challenges in legal applications. Legal work is rife with nuances and complexities that do not conform to predictable patterns, leading to the saying, "no one case is the same." Because AI often lacks all of the context and specificity required for nuanced legal scenarios, it may fail to provide accurate or relevant information. Moreover, general AI models like GPT, Claude, or Gemini were not specifically designed with legal data in mind. Their responses often lack the depth and precision required for thorough legal analysis and advice. Additionally, most AI models are not connected to real-time internet, limiting their ability to provide up-to-date information.

​

Beneficial Uses of AI

1. Information Extraction: AI excels at sifting through vast amounts of data to extract relevant information. Notably, the best way to ensure that the text generation is accurate is to provide the AI, such as GPT, with a specific list of pieces of information you want to extract from a given document or suite of documents. In the prompt, include this list and ask for the exact wording of the text if relevant. For example, I used this approach to extract information from a large suite of documents in an immigration application. By asking for the exact information in a list of bullet points and instructing the AI to indicate where the information was not available by stating "not yet provided," I could easily identify what information had yet to be provided by the client. This method allowed me to verify the accuracy of the AI's responses effectively. The next step would be to ask the AI to draft an email listing the information that appears not provided, ensuring you always check the accuracy of the initial prompt. (Please note that this exercise was done using Donna AI and not any open-source GPT due to the sensitive personal data involved). 

​

​​

​

​

​

​

​

​

​

​

​

​

​

​

​​

​

2. Summarization: One of the more practical applications of AI is its ability to summarize lengthy documents. Specifying what is and is not important in the requested summary is crucial. Using bullet points and examples in the prompt can guide the AI effectively. For instance, an example prompt could be:

 

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

3. Brainstorming and Idea Generation: AI can assist in brainstorming sessions by generating ideas and suggesting alternative approaches. Additionally, you can role-play with AI to predict the counterclaims the other side might use against you and prepare a defense to the same. This can be particularly beneficial when exploring new legal strategies or considering various facets of a complex case.

 

4. Administrative Tasks: AI can efficiently handle administrative duties such as compiling information into tables or charts. Specifically, you can ask AI to provide certain text in a table format and assist with Excel formatting.

 

5. Multilingual Translation: In a globalized world, AI's ability to translate documents and communications into multiple languages is invaluable. However, this should be limited to fairly straightforward language translation and not used for complex legal or medical reports or contracts. You can also ask AI to review a text and highlight common errors or suggestions on where the author could improve their language skills.

​

Limitations of AI

1. Complex Legal Research: Many examples of hallucinations have occurred while testing AI in legal research. This issue is tied to the complexity and volume of documents required in conducting comprehensive legal research. Currently, this is one of the weakest use cases for AI like GPT. I am confident this will improve in the future; however, for now, I don't believe AI is sufficiently reliable for in-depth legal research. 

​

2. Complete understanding of the Context to provide comprehensive advice: Most legal cases and transactions involve multiple parties, various pieces of legislation, and often span different jurisdictions. AI, as it stands today and from what I've seen, does not have the capability to take in all of this context and provide comprehensive and complete advice. While AI can answer specific questions about, for instance, the Spanish Civil Code, it struggles with comparing a particular situation under the Spanish Civil Code with multiple cases in jurisprudence under Irish law. This level of comparative analysis across different legal systems and contexts is beyond the current capabilities of AI. Although advancements may come in the future, the technology is not there yet for now.

​

3. Complex Multi Contract Review: While AI can handle straightforward and specific contract reviews effectively, it struggles with reviews across multiple contracts. In my experience, this process requires very specific prompting. AI does not inherently understand that similar clauses can be termed differently across various agreements. Therefore, a comprehensive review needs to be conducted agreement by agreement, with significant guidance from the prompter to ensure accuracy. However in this case, I believe that sooner rather than later, AI will achieve reasonable accuracy.

​

4. Calculating Numbers: Calculating and analyzing numbers is an area where AI often struggles, especially when compared to its capabilities with text. While there are some specialized AI products designed for analyzing and organizing numbers, my focus here is on general AI models like GPT, Claude, or Bard. In my experience, especially when the input provided to the AI is primarily numerical with very little text or context, the output has been notably weak.

​

To be more specific, I fed GPT with my bank statements and asked it to help me organize my monthly incomes and expenditures for the purpose of drafting an affidavit of means, a common document required in family law proceedings. Despite experimenting with the prompt extensively, I did not manage to get the desired outcome, which was a detailed breakdown of my monthly incomes and expenditures categorized correctly. This example highlights the limitations of general AI in handling tasks that involve complex numerical data and require precise categorization and analysis. I am open to working with anyone on improving my prompting techniques to achieve the desired AI output. If you have any suggestions or would like to collaborate, please let me know.

​

Recommendations

In conclusion, while AI offers remarkable capabilities that can greatly enhance productivity and efficiency in legal practice, it is essential to understand its limitations and use it judiciously. Here are some recommendations for effectively integrating AI into your workflow:

​

1. Use AI as an Assistant:
Utilize AI as a helpful assistant while you go about your work. Break down your workflow into specific tasks and leverage AI to speed up those tasks where possible.

​

2. Check Every Single Line:
When using AI-generated content, meticulously check every single line. Never send out AI-generated text without reviewing it word for word to ensure accuracy and appropriateness.

​

3. Don't Let Mistakes Put You Off:
AI can make mistakes, but don't let that discourage you. Always reiterate and improve upon the initial output you receive. Continuous refinement can lead to better results.

​

4. Make Using AI Second Nature:
Integrate AI into your daily routine to the point where it becomes second nature. Dictate your prompts, create as many first drafts of emails or documents as possible, and brainstorm with it on both personal and professional matters. The more you use AI, the better you will become at it, and the more efficient you will be in all your tasks.

​

By following these recommendations, you can harness the power of AI to enhance your efficiency and productivity while remaining vigilant about its limitations and ensuring the quality of your work.