Conversations on Strategy Podcast – Ep 21 – C. Anthony Pfaff and Christopher J. Lowrance – Trusting AI: Integrating Artificial Intelligence into the Army’s Professional Expert Knowledge

Integrating artificially intelligent technologies for military purposes poses a special challenge. In previous arms races, such as the race to atomic bomb technology during World War II, expertise resided within the Department of Defense. But in the artificial intelligence (AI) arms race, expertise dwells mostly within industry and academia. Also, unlike the development of the bomb, effective employment of AI technology cannot be relegated to a few specialists; almost everyone will have to develop some level of AI and data literacy. Complicating matters is AI-driven systems can be a “black box” in that humans may not be able to explain some output, much less be held accountable for its consequences. This inability to explain coupled with the cession to a machine of some functions normally performed by humans risks the relinquishment of some jurisdiction and, consequently, autonomy to those outside the profession. Ceding jurisdiction could impact the American people’s trust in their military and, thus, its professional standing. To avoid these outcomes, creating and maintaining trust requires integrating knowledge of AI and data science into the military’s professional expertise. This knowledge covers both AI technology and how its use impacts command responsibility; talent management; governance; and the military’s relationship with the US government, the private sector, and society.

Read the monograph: https://press.armywarcollege.edu/monographs/959/

Keywords: artificial intelligence (AI), data science, lethal targeting, professional expert knowledge, talent management, ethical AI, civil-military relations

Episode transcript:

Trusting AI: Integrating Artificial Intelligence into the Army’s Professional Expert Knowledge

Stephanie Crider (Host)

You’re listening to Conversations on Strategy. The views and opinions expressed in this podcast are those of the authors and are not necessarily those of the Department of the Army, the US Army War College, or any other agency of the US government.

Joining me today are Doctor C. Anthony Pfaff and Colonel Christopher J. Lowrance, coauthors of Trusting AI: Integrating Artificial Intelligence into the Army’s Professional Expert Knowledge with Brie Washburn and Brett Carey.

Pfaff, a retired US Army colonel, is the research professor for strategy, the military profession, and ethics at the US Army War College Strategic Studies Institute and a senior nonresident fellow at the Atlantic Council.

Colonel Christopher J. Lowrance is the chief autonomous systems engineer at the US Army Artificial Intelligence Integration Center.

Your monograph notes that AI literacy is critical to future military readiness. Give us your working definition of AI literacy, please.

Dr. C. Anthony Pfaff

AI literacy is more aimed at our human operators (and that means commanders and staffs, as well as, you know, the operators themselves) able to employ these systems in a way that not only we can optimize the advantage these systems promise but also be accountable for their output. That requires knowing things about how data is properly curated. It will include knowing things about how algorithms work, but, of course, not everyone can become an AI engineer. So, we have to kind of figure out at whatever level, given whatever tasks you have, what do you need to know for these kinds of operations to be intelligent?

Col. Christopher J. Lowrance

I think a big part of it is going to be also educating the workforce. And that goes all the way from senior leaders down to the users of the systems. And so, a critical part of it is understanding how best AI-enabled systems can fit in, their appropriate roles that they can play, and how best they can team or augment soldiers as they complete their task. And so, with that, that’s going to take senior leader education coupled with different levels of technical expertise within the force, especially when it comes to employing and maintaining these types of systems, as well as down to the user that’s going to have to provide some level of feedback to the system as it’s being employed.

Host

Tell me about some of the challenges of integrating AI and data technologies.

Pfaff

What we tried to do is sort of look at it from a professional perspective. And from that perspective, so I’ll talk maybe a little bit more later, but, you know, in many ways there are lots of aspects of the challenge that aren’t really that different. We brought on tanks, airplanes, and submarines that all required new knowledge that not only led to changes in how we fight wars and the character of war but corresponding changes to doctrine organizational culture, which we’re seeing with AI.

We’ve even seen some of the issues that AI brings up before when we introduce automated technology, which, in reducing the cognitive load on operators introduces concerns like accountability gaps and automation biases that arise because humans are just trusting the machine or don’t understand how the machine is working or how to do the process manually, and, as a result, they’re not able to assess its output. The paradigm example of that, of course, is the USS Vincennes incident, where you have an automated system. Even though there was plenty of information that it was giving that should have caused a human operator not to permit shooting down what ended up being a civilian airliner. So, we’ve dealt with that in the past. AI kind of puts that on steroids.

Two of the challenges that I think that are unique to AI, with data-driven systems, they actually can change in capabilities as you use them. For instance, a system that starts off able to identify, perhaps, a few high-value targets, over time, as it collects more data, gets more questions. And as humans see patterns, or as a machine identifies patterns, and humans ask the machine to test it, you’re able to start discerning properties of organizations, both friendly and enemy, you wouldn’t have seen before. And that allows for greater prediction. What that means is that the same system, used in different places with different people with different tasks, are going to be different systems and have different capabilities over time.

The other thing that I think is happening is the way it’s changing how we’re able to view the battlefield. Rather than a cycle of Intel driving OPS, driving Intel and so on, with the right kind of sensors in place, getting us the right kind of data, we’re able to get more of a real-time picture. The intel side can make assessments based on friendly situations, and the friendly can make targeting decisions and assessments about their own situation based on intel. So, that’s coming together in ways that are also pretty interesting, and I don’t think we fully wrestled with yet.

Lowrance

Yeah, just to echo a couple of things that Dr. Pfaff has alluded to here is that, you know, overarching, I think the challenge is gaining trust in the system. And trust has really earned. And it’s earned through use is one aspect. But you’ve got to walk in being informed, and that’s where the data literacy and the AI literacy piece comes in.

And as Dr. Pfaff mentioned, these data-driven systems, generally speaking, will perform based on the type of data that they’ve been trained against and those types of scenarios in which that data was collected. And so, one of the big challenge areas is the adaptation over time. But they are teachable, so to speak. So, as you collect and curate new data examples, you can better inform the systems of how they should adapt over time. And that’s going to be really key to gaining trust. And that’s where the users and the commanders of these systems need to understand some of the limitations of the platforms, their strengths, and understanding also how to retrain or reteach to systems over time using new data so that they can more quickly adapt.

But there’s definitely some technical barriers to gaining trust, but they certainly can be overcome with the proper approach.

Host

What else should we consider, then, when it comes to developing trustworthy AI?

Pfaff

We’ve kind of taken this from the professional perspective, and so we’re starting with an understanding of professions that a profession entails specialized knowledge that’s in service to some social good that allows professionals to exercise autonomy over specific jurisdictions. An example, of course, would be doctors and the medical profession. They have specialized knowledge. They are certified in it by other doctors. They’re able to make medical decisions without nonprofessionals being able to override those.

So, the military is the same thing, where we have a particular expertise. And then the question is, how does the introduction of AI affect what counts as expert knowledge? Because that is the core functional imperative of the profession—that is able to provide that service. In that regard, you’re going to look at the system. We need to be able to know, as professionals, if the system is effective. It also is predictable and understandable. I am able to replicate results and understand the ones that I get.

We also have to trust the professional. That means the professional has to be certified. And the big question is, as Chris alluded to, in what? But not just certified in the knowledge, but also responsible norms and accountable. The reason for that is clients rely on professionals because they don’t have this knowledge themselves. Generally speaking, the client’s not in the position to judge whether or not that diagnosis, for example, is good or not. They can go out and find another opinion, but they’re going out to go seek another profession. So, clients not only need to trust t...