Asking the Wrong Questions

Maybe when it comes to the chat bots and art bots and other such creations, we’re asking the wrong questions.

I mean, if a chat bot can pass the bar exam, the question shouldn’t be “can a chat bot practice law” but rather “does the bar exam do a good job of determining whether someone would make a good lawyer.”

Having taken a bar exam, I can assure you it’s primarily a hazing ritual. I’m sure the chat bots do very well on the multistate multiple guess portion of the exam, which requires you to memorize vast amounts of information, much of which is not relevant to actual practice.

My experience with bar exams is out of date, but when I took it, we had to learn all the old common law (based on British law) definitions of criminal behavior. These were no longer in use in Texas (where I took the bar) or in any other state that had adopted a modern penal code.

I bet a chat bot is hell on wheels at stuff like that, but I suspect a bot lawyer would not know what to do in a situation where its client was before a judge for revocation of probation (on a felony drug charge) and the judge, in the middle of ranting at its client, gave it a huge wink.

I’m not even sure how a chat bot would know about the wink, but assuming a bot could see it, I suspect it wouldn’t know it was a signal that the judge wasn’t going to revoke probation.

The time it happened to me, I knew what it meant. In fact, I knew from the moment the judge started ranting that he wasn’t going to send my client to prison. He winked at me because I was a young lawyer and he wasn’t sure I understood yet that he was yelling at my client in lieu of revoking probation.

The real practice of law is about subtleties. I suspect algorithm-driven software fueled by large language models is very useful in plowing through reams of documents and will get better, but it’s going to be crap at the negotiating table or in the courtroom where you have to read people as well as put the right information before them at the right time.

Continue reading “Asking the Wrong Questions”