When Resources Are Limited and Stakes Are High: AI in Healthcare Decision-Making

Healthcare professionals are often required to make critical decisions under less-than-ideal conditions. Information may be incomplete, resources may be limited, and time is rarely on their side. These challenges become even more pronounced in environments such as disaster response, battlefield medicine, and mass casualty incidents. They are also common in rural and other resource-limited healthcare settings, where access to personnel, equipment, or specialized expertise may be constrained.

As organizations look for ways to support decision-making in these high-pressure situations, artificial intelligence is emerging as a potential tool. From diagnostic imaging to patient monitoring, AI is becoming increasingly integrated into healthcare workflows. However, adoption raises important questions around four areas:

• Reliability: Does the system perform consistently and accurately across diverse patients and settings?
• Bias: Have potential disparities in training data or model behavior been identified and mitigated?
• Transparency: Can clinicians understand why the AI made a recommendation?
• Clinical trust: Are healthcare providers confident enough in the system to act on its guidance?

These are not abstract concerns. Healthcare decisions carry life-altering consequences, and AI systems that introduce bias, obscure their reasoning, or perform inconsistently across populations can cause real harm. Responsible AI adoption in medicine requires rigorous validation, continuous oversight, and sustained human involvement.

Unique Challenges and Opportunities in High-Pressure Settings

Standard clinical environments present challenges for AI adoption, but high-pressure settings such as disaster response, battlefield medicine, and mass casualty events take those challenges to an extreme. In these scenarios, the conditions that AI systems are typically tested under (well-resourced hospitals, stable connectivity, experienced multidisciplinary teams) simply do not exist.

What makes these environments different

• Resource scarcity: Limited personnel, equipment, medication, and time force rapid prioritization decisions.
• Information uncertainty: Patient histories may be unavailable; vital sign data may be incomplete or unclear.
• Moral complexity: Triage inherently involves difficult trade-offs about who receives care first when not everyone can be treated.
• Provider fatigue and cognitive load: Human decision-makers in these high-stress settings are themselves under extreme pressure.

While these challenges are often associated with battlefield medicine and mass casualty events, they are not limited to emergency scenarios. They also occur during natural disasters and in healthcare settings where personnel, equipment, or specialized expertise may be constrained. Clinicians in these environments must often make critical decisions with limited information and available resources.

These are also the conditions where well-designed AI decision support can have the greatest impact. A system that processes physiological data rapidly, surfaces patterns that fatigue might cause a provider to miss, and brings consistency to high-stakes decisions can genuinely improve outcomes when seconds matter.

But the stakes of getting it wrong are equally high. AI tools in these environments must be held to a higher standard of explainability and alignment with human judgment.

Applying Human-Centered AI to Medical Triage

Kitware has been working directly on AI tools designed for these conditions through DARPA’s In the Moment (ITM) program, which explores how AI can reliably support human decision-makers in complex, high-stakes situations where no single right answer exists.

Kitware participated in Phase 1 of ITM, focusing on military medical triage in resource-constrained environments, where speed, accuracy, and trust are paramount. Kitware was awarded an $11.5 million, multi-year contract to develop algorithmic decision-makers that align with human judgment, enabling explainable, human-centered AI-assisted decisions.

Traditional AI models often fail to earn trust in high-stakes domains. Accuracy alone is not enough when decisions involve trade-offs, incomplete information, and contextual reasoning.

Conventional systems typically optimize for a single metric, such as survival probability, but real-world triage requires weighing multiple priorities and adapting to the reasoning style of the person making the call. DARPA needed an approach built on transparency and human alignment, not a black box that simply outputs a recommendation.

Kitware developed algorithmic decision-makers, built on large language models, that emulate human goals, intentions, and values by aligning with Key Decision-Making Attributes (KDMAs). The approach integrates explainability, alignment with human values, and adaptability, so that AI decisions are interpretable and trustworthy rather than opaque.

The system separates evaluation from decision-making: it first assesses multiple possible courses of action against these KDMAs, then produces a final recommendation with reasoning that a human can review and challenge. This lets the AI support human judgment rather than override it, adapting in real time to different decision-making styles. Realistic testing used Kitware’s open source Pulse Physiology Engine to generate synthetic patient profiles across a broad range of scenarios and injury types.

The lessons learned through ITM also highlight a broader question: how should organizations determine whether AI is appropriate for a given healthcare task?

Read more: Explainable AI in Action: DARPA ITM Phase 1 Contributions

Responsible AI: Evaluating Whether AI Is Right for a Task

Not every healthcare task is appropriate for AI assistance. The enthusiasm around AI capabilities can create pressure to deploy tools before they have been sufficiently validated for high-stakes clinical use. Responsible adoption requires asking harder questions up front.
Drawing on the work done through ITM, here are the criteria organizations should apply when evaluating whether an AI system is appropriate for a given task:

1. Can it explain itself?
   Black-box outputs are not sufficient in settings where the reasoning behind a decision matters as much as the decision itself. The system must be able to explain its recommendations in terms clinicians can evaluate and challenge.
2. Does it reflect human values?
   A system that optimizes for a single metric, such as survival probability, may not capture the full complexity of real-world decisions. Human alignment means reflecting the values, priorities, and reasoning patterns of the people the system is designed to support.
3. Has it been validated across the populations it will serve?
   Performance on benchmark datasets does not guarantee performance in the field. Systems must be tested across the patients and scenarios they will actually encounter.
4. Can its behavior be examined after the fact?
   Audit trails and transparent architectures are necessary for accountability. Organizations need to be able to review how and why the system made a given recommendation.
5. Is the human always in control?
   AI in high-stakes settings should be designed for human-in-the-loop operation. The human must always be able to override the system’s recommendation.
6. Are the validation requirements proportional to the stakes?
   A tool supporting administrative scheduling requires a different bar than one supporting triage in a mass casualty event. The higher the stakes, the more rigorous the evaluation needs to be.

Looking Ahead

AI is already playing a role in emergency medicine, battlefield triage, and disaster response. The question is not whether it will be used, but how it will be built, evaluated, and governed.
Kitware’s work on DARPA’s In the Moment program reflects a deliberate answer to that question: AI systems that are transparent, human-aligned, and designed to support rather than replace human judgment. As this technology matures and moves toward broader deployment, the frameworks developed through this program provide a foundation for responsible adoption.

The future of AI in healthcare will not be defined solely by model performance. It will be shaped by how effectively these systems earn trust, support human decision-making, and remain accountable when the stakes are highest. As organizations continue evaluating AI for healthcare applications, transparency, human oversight, and rigorous validation will remain essential to responsible adoption.