- within Compliance topic(s)
By now, we’re all familiar with the travails of lawyers who use gen AI to draft court papers, but fail to check them and file briefs including fake cases, as well as other significant maladies, resulting from AI hallucinations and being sanctioned for their carelessness. Out of Brazil, however, comes a new twist: according to media reports, a Brazilian court recently sanctioned lawyers for intentionally including hidden messages, known as “prompt injection,” in a court filing, apparently designed to deceive the court and increase the chances of their success.
“Prompt injection” is, essentially, the practice of including hidden text in a document that gives a command to a gen AI program when that document is fed into it. For example, someone submitting a cover letter for a job might include hidden text commanding any AI program reviewing the cover letter to move the application materials to the top of the pile, or to respond if prompted that the applicant is the best suited regardless of the criteria.
In the Brazilian case, the media reports indicate that the lawyers included white text on a white background that essentially attempted to direct gen AI not to contest the lawyers’ petition. The Brazilian court apparently, and luckily, used a sophisticated gen AI program that flagged the text and refused to process the lawyers’ hidden command. It is not clear whether all gen AI programs – particularly free programs used by the general public – would detect a prompt injection, however.
Although this story may unlock a new nightmare for many courts and lawyers, the chances of lawyers repeating this conduct are low. Lawyers are bound by ethical rules that should discourage this type of behavior, and which will lead to serious consequences if undertaken. The American Bar Association and other bar associations have issued ethics guidelines and opinions regarding gen AI and the practice of law, but it remains to be seen whether any rules of professional conduct will be updated to address new issues created by the rise of gen AI. In any event, courts are going to come down very hard on attorneys who attempt any such conduct.
A bigger concern may be the use of prompt injections by non-lawyers who are not bound by ethical rules, and in non-court related matters. What if a business competitor sends a company a letter with a prompt injection that instructs a gen AI program to provide misleading answers to the company or to somehow release confidential information? The competitor may be subject to civil liability if caught, but it could wreak significant havoc in the meantime.
The threat of a prompt injection may not yet be significant enough to keep lawyers and companies up at night. But stories like the Brazilian incident underscore the need to stay on top of the latest developments with respect to gen AI, and to use secure, confidential, and sophisticated gen AI programs that are able to flag prompt injections. The use of gen AI will continue to evolve in ways—both positive and negative—that we cannot yet predict. Staying up to speed is critical.
The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.
[View Source]
