#559 Who's Driving? What AI in Autonomous Cars Teaches Business Leaders About Risk, Trust and Control | George Pillari interviewed by Niels Brabandt
Who's Driving? What AI in Autonomous Cars Teaches Business Leaders About Risk, Trust and Control
As artificial intelligence moves from advisory tool to autonomous decision-maker, the question facing every executive is no longer whether to adopt AI, but how much control to hand over, and who answers for the outcome when something goes wrong. In this week's episode of The Leadership Podcast, Niels Brabandt EMBA MBA MSc is joined again by AI and technology commentator George Pillari, publisher of The Cautionary newsletter, to unpack what the slow, careful rollout of autonomous vehicles reveals about deploying AI responsibly inside any organisation.
Three Words Every Executive Should Know
George Pillari introduces a framework that reaches far beyond the automotive industry: human in the loop, human on the loop, and human out of the loop. In the loop describes a person actively directing an AI system, comparable to how most professionals currently use tools such as Claude. On the loop describes a system operating independently while a human monitors and can intervene, similar to aircraft autopilot. Out of the loop, where the system acts without real-time human oversight, is where the difference between convenience and genuine risk begins, whether the system is steering a car or approving a financial transaction.
Accountability Does Not Disappear, It Moves
Niels Brabandt presses Pillari on the uncomfortable question that follows every automated decision: who is responsible when an autonomous system causes harm. Pillari's answer, that this is becoming one of the defining ethical debates of the next decade, applies directly to any executive introducing AI into hiring, lending, medical triage or safety-critical operations. The lesson for leaders is that automation does not remove accountability from the organisation, it relocates the decision earlier, into the design and training of the system itself.
The Danger Hiding in Edge Cases
Pillari explains why autonomous vehicles have not scaled as quickly as early predictions suggested: edge cases, the rare situations a system has never been trained to handle. An AI, he notes, only knows what it has been able to consume and learn, and unfamiliar scenarios, from a police standoff to a rare calendar date, can leave even sophisticated systems without a safe response. For business leaders, the parallel is direct. Any AI system performs reliably within the data it has seen and unpredictably outside it, which makes rigorous testing of edge cases, not just average performance, the real measure of readiness.
Insurance Is the Quiet Gatekeeper
One of the most striking points in the conversation concerns an industry rarely discussed alongside AI: insurance. Brabandt raises the question directly, noting that without insurers willing to underwrite the risk, adoption cannot move forward regardless of how capable the technology becomes. Pillari's assessment is that autonomous vehicle manufacturers are currently absorbing much of that risk themselves, because insurers lack the historical data to price it accurately. Executives evaluating AI investments should take the same view: technical readiness means little if the organisation's risk, compliance and insurance functions are not equally prepared.
Trust Is Built Long Before the Technology Is Ready
The conversation closes on aviation, where autonomous systems already fly the vast majority of every commercial flight, yet passengers still expect a human pilot in the cockpit. Pillari predicts that autonomous cargo flight will arrive before mainstream autonomous passenger vehicles, precisely because it removes the trust problem rather than solving it. For leaders rolling out AI internally, the lesson is the same: adoption succeeds not only when the technology works, but when the people affected by it trust the system enough to rely on it.
The Bottom Line for Decision-Makers
Autonomous vehicles are not simply a transportation story, they are a preview of how any organisation will need to manage AI that acts without constant supervision. Niels Brabandt and George Pillari's conversation offers a working vocabulary, human in the loop, on the loop and out of the loop, for boards and executive teams to classify their own AI initiatives and decide, deliberately, how much control to hand over and where accountability must remain firmly human.
Niels Brabandt
---
More on this topic in this week's videocast and podcast with Niels Brabandt: Videocast / Apple Podcasts / Spotify
For the videocast’s and podcast’s transcript, read below this article.
Is excellent leadership important to you?
Let's have a chat: NB@NB-Networks.com
Contact: Niels Brabandt on LinkedIn - https://www.linkedin.com/in/nielsbrabandt
Website: www.NB-Networks.biz
Podcast and Videocast Transcript
Niels Brabandt EMBA MBA MSc
AI is everywhere, and it's in your car as well. And now you probably think, well, how far can it go, and can it go on its own, and where will the developments go? We have an expert on the matter with us here today. Hello, and welcome back by popular demand, George Pillari.
George Pillari
Nice to be here.
Niels Brabandt EMBA MBA MSc
Thank you very much for coming back again.
Niels Brabandt EMBA MBA MSc
AI in autonomous cars: when we talk about autonomous cars, many people wonder, where are we? We hear that there is Waymo in San Francisco and in a couple of other cities. Still, people say as soon as you have AI in other cars by other vendors, they always say, you need to keep your eyes on the road, you need to keep your hands on the steering wheel.
Niels Brabandt EMBA MBA MSc
So, first question here: where are we at the moment with AI in cars in general?
George Pillari
Sure. If you just go back to the beginning, or almost the beginning, of electronic vehicles even, right, which world this is happening in. This was 25 years ago, where Elon Musk demoed a sample for the founders of Google, right, Larry Page and Sergey Brin, and it didn't really work. It went like 10 miles an hour or something, and they still invested with them, okay?
George Pillari
So anyway, we've come a long way where you have all these features now built into electric cars that can do things for you. Now, there's a couple of important terms to remember here that we're going to be learning about and hearing about for the rest of our lives, right? It's human in the loop, human on the loop, and human out of the loop.
Niels Brabandt EMBA MBA MSc
What does that mean?
George Pillari
It's exactly, it's exactly what it sounds like, okay? When you're in the loop, when you're the human in the loop, right, the AI is you're controlling the AI, giving it direction. Like most, most of us use Claude and things like that.
George Pillari
When you're human on the loop, you're letting the AI do the work, and you're monitoring things. Think of the, like, autopilot on an aircraft, right? That's, that's human on the loop, right? You handed control over, but you can intervene.
Niels Brabandt EMBA MBA MSc
Okay.
George Pillari
Okay? Now, human.
Niels Brabandt EMBA MBA MSc
So this is compatible, for example, something like Claude Cowork, where you say, do everything, and as soon as Claude has a question, it's just like, hey, what about this one? What should I do here?
George Pillari
Right. Okay.
Niels Brabandt EMBA MBA MSc
George Pillari
Correct. Right, so this human, human in the loop, human on the loop, and now human out of the loop. Human out of the loop is where there's the most controversy, all right? And this is where the autonomous vehicles come in. Same thing for weapons in the military.
Niels Brabandt EMBA MBA MSc
Yes.
George Pillari
All right? Do you turn over control, right, to an autonomous system? Because who's responsible when the autonomous car makes a mistake, right, and it, and it, you know, causes an accident? Who's responsible?
Niels Brabandt EMBA MBA MSc
And also, also, let's, let's face one situation. Someone has to program the AI. So let's say when the AI system knows that the accident can't be prevented, do you run into that group of pensioners or that group of younger people? So who wants to program this?
George Pillari
That's right. So we're going to run into really deep questions on ethics as these things proliferate, become, become mainstream.
George Pillari
Now, with electronic, with autonomous vehicles, we're starting to see, you know, some signs of life, right? We're starting to see new cities being announced, and even new countries, right, every couple of weeks.
George Pillari
You ask, how come it's taken so long, right? If you have a mousetrap, can't we just use it everywhere, right? And that, that goes to another really important concept we're going to hear about again for the rest of our lives, okay, which is called edge cases, all right?
George Pillari
And an edge case is exactly what it sounds like. Remember, AI gets trained on tons and tons of data.
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
Okay? It just, just consumes data. Now, but how many times do you think the AI has seen a sunrise on February 29th, right? Like, that's, that's not something it sees a lot.
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
So it may not know what to do in that circumstance. If you go back, some of our more seasoned listeners here will remember the Y2K crisis, where everybody was concerned that what's going to happen when.
Niels Brabandt EMBA MBA MSc
I remember.
George Pillari
Computers don't know what to do. Because we used to.
Niels Brabandt EMBA MBA MSc
I was sitting there as well. Exactly. We were at the computer. Three, two, one. Exactly.
George Pillari
Yes. So, you know, you have these, these edge cases, which are extremely dangerous because, you know, you have the AI can't know everything. It only knows what it's been able to consume and learn. So the thing you hear about lately is where an autonomous vehicle just drives right into the middle of a crime scene.
Niels Brabandt EMBA MBA MSc
I remember. It was in a police standoff. I read the article about that. Yes.
George Pillari
Yes. So, you know, are we there yet? Sort of. I think in a controlled environment where, you know, you're going to run that vehicle in a 10-block radius, and you can train it on everything, or most everything you're going to see in that 10-block radius, it's pretty useful.
George Pillari
But the, you know, the example I like to use is sometimes the AI is way better than a human, right? Many times it's not. So here's, here's a good one for you. You're, the autonomous vehicle is moving along, and all of a sudden a bicycle rider jumps in front of the, in front of the car, all right? If you're driving the car on your own, you're instinctively going to swerve left or you're going to swerve right. Yeah. And
Niels Brabandt EMBA MBA MSc
George Pillari
hit your brakes. So if you go left, you're going into head-on traffic. If you go right, you go into a crowded sidewalk of pedestrians, all right? So there's no good answer here.
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
And so what happens is the AI is programmed. This is done years ago, right? So the AI doesn't have to think. As a human, you're going to take, I don't know, half a second, you know, some fraction, even if you're an incredible, you know, you have incredible reflexes, to process what's going on in front of you and then take an action.
George Pillari
The AI doesn't need to process it. It knows what it saw, and it's built into its programming, and in a millisecond or two, it does what it's programmed to do. And in this case, the programming is generally to continue the trajectory and minimize risk, is the term they use. So don't swerve, because you're going to create new problems that are unknowns. You have one known in front of you, that's a bicycle rider. Frankly, it was their fault for jumping in front of you.
George Pillari
But so what they try to do is apply maximum braking pressure and minimize the impact. So imagine sitting around the conference room with the engineers five years ago, ten years ago, before these things were programmed in, and you run through these scenarios. You know, so they sit there, well, the best answer is to, you know, cut your losses and not create new problems. And that's where the AI is really powerful, all right? Really powerful, way more powerful than a human.
George Pillari
You want the, you want the AI making that decision. But, you know, what if it's your neighbor on the bike? You know? So it's, you run again, this whole world of AI ethics is going to be a booming field for the next generation.
Niels Brabandt EMBA MBA MSc
Absolutely. And of course, one question I need to have here, especially with autonomous cars, is there's one big player who needs to have a huge chunk taking the risk in this. This is insurance companies.
Niels Brabandt EMBA MBA MSc
Because when we say we like to have this in place, if that player says, no, no, we're not moving anywhere. So is there any kind of, any kind of research going on how to work with these companies? Because when they, at the end of the day, say no to whatever we plan here, nothing's going to move forward.
George Pillari
That's a great question, and not one we hear a lot. And I can give you some informed speculation on this. I think what's going on is the AV makers, autonomous vehicle makers, are subsidizing the insurance. Or they're, or they're, or they're eating it themselves, okay? Because these insurance is like AI, right?
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
They look at, they look at the pool. Hey, you're this age, here's your non-smoker. They run all the numbers on people in that cohort, and they tell you what your risk of death is, or your risk profile as a driver.
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
So I, I think in the background, the AV companies are, are the ones subsidizing or just eating all that risk. Maybe they have some, some stop-loss coverage at a very high level, because there's no data for these insurers to underwrite.
Niels Brabandt EMBA MBA MSc
Yeah. Exactly. Exactly.
George Pillari
It's a great, it's a great question.
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
Really interesting topic. I'm going to research that one.
Niels Brabandt EMBA MBA MSc
Yeah. Thank you. One more question I have. When we now say, because we now see, especially in London and the UK, we see Waymo cars going all over the place. And of course, people say, hey, it's great when you go around here and you measure every single square nanometer of the city.
Niels Brabandt EMBA MBA MSc
But hear the news. This is London. This city has got, you, you drive around the block and you come around and it will have changed from there. We will have a construction site here, which is here today, there tomorrow.
Niels Brabandt EMBA MBA MSc
There will be probably people walking here a lot from Monday to Friday, not at all on Saturday and Sunday, other areas, vice versa. So how good is AI at the moment from being pre-trained, which is great, but how great is AI so far in autonomous cars to deal with situations that cannot be pre-trained because they simply happen?
George Pillari
That's, that's where the deadly problems arise, all right? When there has not been the training and it doesn't know what to do, okay? And, you know, many times it's like a child, I like to say, who goes to the zoo and sees an enormous ape or something, right? Or a lion. The first time they see it, they, wow, they don't know what to do. They know it may be dangerous just instinctually, but, so the AI, that, that's the biggest hole, okay?
George Pillari
And that's why, that's why AVs have not proliferated with, with the kind of gusto we would have expected with the new technology.
George Pillari
Frankly, it's going to be easier to have autonomous planes than it is for cars. Because you can send up a cargo plane and it has a lot of room to roam.
Niels Brabandt EMBA MBA MSc
Yeah.
George Pillari
You know? And you can control it just the same way you control, you know, an RV, I'm sorry, an electronic, you know, toy, right? Controls. But you're going to see autonomous airflight, probably for cargo, probably sooner than you see real mainstream AV usage.
Niels Brabandt EMBA MBA MSc
Yeah. And also, cargo planes take humans out of the equation. Because I know that even if you have the most perfect AI in a plane, you still need to convince people who booked the ticket to go on there and say, look, there is no human steering this, but you're sitting in the back, and when something goes wrong, well, we know where it ends.
Niels Brabandt EMBA MBA MSc
So with cargo, that might be a lot easier. Of course, you have to mitigate the risk of the plane not crashing down, but probably this will be, because that would, of course, be a huge shift in how cargo in the air would work in general, globally.
George Pillari
Yes. You know, you know, people don't realize this, and it might shock them, but in a typical three-hour flight, the pilot only hand-operates the airplane for three or four minutes. The rest of it's handed over to the autonomous systems. You know, aviation's way, way down the path using AI for many years, but the pilot doesn't really fly the plane.
Niels Brabandt EMBA MBA MSc
Yeah. Excellent. So when our companies say, hey, this is a really good topic, so maybe we should actually get George in and hold the key now to our people, or maybe help our people in the future here. The final question I always have to ask, how can people reach out to you?
George Pillari
I'm at thecautionary.com is my newsletter, The Cautionary, and I publish weekly. And right now I'm on my, I don't know, about three or four months of weekly AI posts on things like autonomous vehicles and other topics.
Niels Brabandt EMBA MBA MSc
Perfect. I think these are the perfect final words. You see, AI is here, and it's here to stay, and we will keep track with how autonomous cars will show us the future. George Pillari, thank you very much for your time.
George Pillari
Yeah. Thank you.