Building AI That Humans Can Trust: DARPA’s In the Moment Program

Defense Advanced Research Projects Agency In the Moment (ITM) program explores how AI can reliably support human decision-makers in complex, high-stakes environments. Kitware participated in Phase 1, focusing on military medical triage in resource-constrained settings, where speed, accuracy, and trust are paramount. The program’s broader mission is to create AI systems that not only make accurate recommendations but also reflect human reasoning, values, and priorities. Phase 2 extends this work to cybersecurity, where decisions require careful balancing of factors such as confidentiality, integrity, and availability.

The Challenge: High-Stakes Decision-Making Under Uncertainty

In high-stakes domains, conventional AI models often fail to earn human trust. Accuracy alone is insufficient when decisions involve ethical trade-offs, incomplete information, and contextual reasoning. For Phase 1 of ITM, DARPA challenged teams to create AI capable of supporting medical triage under extreme uncertainty. Traditional systems typically optimize for a single metric, such as survival probability. Still, real-world triage requires responsible consideration of patient outcomes, adaptation to situational priorities, and alignment with the individual decision-makers’ reasoning styles. DARPA needed an approach that emphasized transparency and human-aligned decision-making rather than a “black-box” model that simply outputs recommendations.

The Solution: Building Human-Aligned AI

DARPA selected Kitware to develop AI systems that embed trust, transparency, and ethical reasoning at their core. A key innovation is the LLM-as-a-Judge framework, where large language models evaluate potential options and score them against human-defined Key Decision-Making Attributes (KDMAs). A regression framework then generates the final recommendation, separating evaluation from choice and providing clear, human-understandable reasoning.

To ensure that the AI could reflect diverse human decision-making, the system incorporated few-shot learning and domain adaptation, enabling it to adapt dynamically to individual decision styles. Realistic testing was achieved using the Pulse Physiology Engine, which generated synthetic patient profiles across a broad spectrum of medical scenarios. This combination of techniques enabled AI to make ethically informed, transparent, and adaptable recommendations aligned with human priorities.

Phase 1: Medical Triage

In Phase 1, the focus was on supporting high-stakes medical triage under uncertainty. The AI:

• Used LLMs as Judges: Evaluated options rather than making decisions directly, generating reasoning statements to improve transparency and reduce bias.
• Adapted to Decision Styles: Learned from human operators via few-shot learning, ensuring flexible alignment.
• Tested in Realistic Simulations: Synthetic patient profiles allowed assessment across diverse, complex scenarios.

As a result of these Phase 1 efforts, AI systems outperformed baseline models, earned higher trust ratings, and demonstrated that human-aligned AI can reliably reflect ethical judgment in sensitive, high-pressure environments.

Phase 2: Cybersecurity Alignment

Phase 2 applies these principles to cybersecurity, where decisions must account for competing priorities, including confidentiality, integrity, and availability. Kitware’s AI now integrates:

• Multi-KDMA Reasoning: Algorithms predict the relevance of each KDMA, allowing the system to evaluate trade-offs rather than optimize for a single attribute.
• Autonomous Agents with Responsible Constraints: Reinforcement learning and LLM-driven agents operate under realistic simulations, adhering to human-defined priorities even under pressure.
• Trust as a Measurable Outcome: DARPA aims to increase the proportion of human decisions delegated to AI from 60% in Phase 1 to 85%, thereby highlighting trust as the central metric.

By building alignment into both architecture and training, Kitware’s AI demonstrates that ethically aligned, trustworthy, and adaptable systems are feasible in diverse high-stakes domains.

Trust, Adaptability, and Broader Impact

Kitware’s human-aligned AI systems consistently outperformed baseline models, earning higher trust ratings from evaluators. By separating evaluation from decision-making and embedding alignment at every step, the AI supports human judgment rather than overriding it, adapting dynamically to individual decision-makers in ethically complex scenarios.

The same principles are now applied in Phase 2 to cybersecurity, demonstrating that this approach is transferable across domains that require rapid, high-stakes decisions. Because Kitware’s tools are open source, organizations can build on this foundation to deploy transparent, human-aligned AI in healthcare, defense, emergency response, corporate strategy, and beyond.

By advancing alignment and explainability, ITM technologies address a core challenge across industries: ensuring AI systems enhance human expertise rather than replace it. Kitware’s open source approach provides a platform the community can adapt, validate, and extend, fostering the safe and accountable deployment of AI across diverse high-stakes environments.

Get in Touch

Explore how Kitware’s human-aligned AI research and tools can help your organization deploy transparent, trustworthy AI. Whether integrating AI into high-stakes workflows or scaling for operational deployment, our solutions make it easier than ever to build systems that earn human trust.

Contact Us

Acknowledgment:
This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8650-23-C-7316. The views, opinions, and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.