By Brenda Bannan, PhD Founder, AI4LD/Professor Emerita, George Mason University
Picture a training exercise: 70 first responders — firefighters, emergency medical technicians, and law enforcement officers — spread across a large arena, responding to an active violence simulation. The scenario is high-fidelity, the stakes feel real, and instructors are watching carefully from their observation positions.
The problem, rarely discussed openly, is that the instructors cannot actually see what is happening — not because they are inattentive, but because the human observational system was never designed to track 70 people across a large building simultaneously. Working memory, attention, and perception have finite capacity, and live simulation environments routinely exceed it. Instructors miss critical coordination failures between agencies, cannot measure time to neutralize a threat or rescue a patient, and when the debrief happens twenty minutes later, it is necessarily driven by memory, impression, and assumption rather than by what actually occurred.
What presents as an observation problem is, more precisely, a learning design problem. The question worth asking is not how to make instructors more attentive, but how to design training systems that give instructors — and the teams they train — access to the behavioral evidence that learning requires. Answering that question is what drove a decade of research at the intersection of learning science, human-centered design, and artificial intelligence.
—
A Decade of Design Research: Learning Science First
This work did not begin with a technology. It began with a question about what it takes to design training systems that produce genuine learning in complex, high-stakes, multi-team environments — and that question has remained the organizing frame through every iteration of the research.
Beginning in 2015, my colleagues and I at George Mason University began embedding cyber-physical systems and IoT sensors into training simulations with emergency responder teams (Gallagher, Bannan, & Lewis, 2017), examining how proximity sensors and behavioral data streams could capture what human observation could not under real training conditions. From there, the research program expanded iteratively: toward wearable devices for multiteam systems learning (Bannan, Dubrow, Dobbins, Zaccaro, Purohit, & Rana, 2019), toward sensor and video data streams for improving situation awareness and team coordination during live exercises (Dubrow & Bannan, 2019), and toward the design of a multimodal analytics system built specifically to improve the quality of emergency response training (Purohit, Dubrow, & Bannan, 2019).
Every iteration was grounded in the same foundational principle: operationalize the learning construct before choosing any technology. What does team situation awareness look like behaviorally in this context? What does effective coordination between agencies look like in real time, under pressure? Only after answering those questions did we ask what data would capture these constructs and how. That sequencing — learning science first, technology second — is what distinguishes AI-enabled learning experience design from the more common pattern of selecting tools and then searching for pedagogical justification afterward.
—
The Framework: AI Affordances and Kolb’s Experiential Learning Cycle
Before describing what we built, it is worth naming the conceptual frame that organized this work, because that frame has implications well beyond emergency response training.
An AI affordance is a specific capability that AI enables which was previously impossible or impractical for human systems alone — not faster or cheaper, but genuinely unavailable through human means. Identifying affordances is not a matter of surveying available tools; it requires asking what AI makes possible that connects to how people actually learn. This question, rather than “what AI tools are available?”, is the productive starting point for any AI-enabled learning experience design.
In our research, the AI affordances we identified mapped directly onto Kolb’s (1984) experiential learning cycle, which provided the theoretical architecture for the human-machine adaptive system we were designing (Bannan, Torres, Purohit, Pandey, & Cockroft, 2020):
– Concrete Experience — first responders engage in live simulation while sensors capture behavioral and environmental data seamlessly and unobtrusively
– Abstract Conceptualization — AI processes multimodal data streams, with machine learning and computer vision extracting behavioral patterns that fall outside the range of human observation
– Reflective Observation — visualized data is presented to teams during the debrief, where trainers and leadership guide interpretation and sensemaking using the analytics dashboard
– Active Experimentation — teams adjust behavior in the next simulation run based on that reflection, while the computational system also learns from each run
What is significant about this architecture is that the learning cycle runs in both directions: the human system and the computational system adapt together across iterations, each informing the other. This is not a content delivery model but a genuine adaptive instructional system — a learning environment that evolves with use rather than remaining static.
—
The Case Study: CitizenHelper-Training
In 2020, our research team deployed the CitizenHelper-training system — an AI-infused multimodal analytics platform — during a full-scale exercise at a large arena on a mid-Atlantic university campus (Pandey, Purohit, & Bannan, 2020).
The participants were more than 70 first responders from fire, EMS, and law enforcement — three agencies that had never collectively trained together before — responding to an active violence incident simulation. The learning challenge was therefore not only tactical but inter-organizational: how do teams that operate under different communication protocols, different chains of command, and different training cultures coordinate effectively under pressure, particularly when they have no prior shared experience to draw on?
What we built:
The CitizenHelper-training system integrated four components into a single architecture. Multimodal data collection drew from 24 sensor pods distributed throughout the arena, each equipped with LiDAR occupancy detectors, blue-force tracking, Wi-Fi indoor positioning, biometric wearables, and environmental sensors monitoring particulate matter, temperature, and carbon dioxide — alongside IP cameras positioned at multiple locations capturing video throughout the exercise. An AI computation engine processed these streams in parallel, with FasterRCNN detecting people across camera feeds and MobileNet classifying actors by role (patient, responder, observer); transfer learning allowed pre-trained models to be fine-tuned on domain-specific data, a critical design decision given the scarcity of labeled emergency training footage. A cloud-based Elasticsearch database provided fast real-time indexing and retrieval of behavioral events, and a Kibana visualization dashboard delivered findings through hierarchical progressive disclosure — broad situational overview first, interactive timeline filtering available on demand — alongside a 3D digital twin of the arena facility that gave all stakeholders a shared operational picture of the building environment.
What AI extracted that human observation could not:
This is where the affordance argument becomes concrete. Human instructors, observing from fixed positions, could not reliably extract time to neutralize the threat, time to rescue patients, distribution of responders versus patients across the arena over time, which responders clustered around the first patients they found while others in a different wing of the building went unattended, or the coordination patterns between agencies at specific critical inflection points — patient handoffs, communication breakdowns, role transitions. The AI engine extracted all of these, not without limitation (environmental false positives, scarce labeled training data), but at a level of fidelity and completeness that no human observation team could sustain at that scale and pace.
—
The Four AI Affordances in Action
Across this research program, four AI affordances emerged as central to enhancing first responder team learning, each connecting to an established body of learning science rather than to the capabilities of any particular technology.
Affordance 1: Multimodal data capture at scale
IoT sensors and computer vision captured behavioral data across 70+ responders simultaneously, directly addressing the working memory constraint that has always limited observation-based training. Human working memory operates within narrow bounds, and live simulation environments regularly produce more behaviorally significant events than any observer can track, integrate, and retain. A multimodal sensor network does not share this limitation, which means, for the first time, that instructors entered the debrief with access to complete behavioral data rather than a sample filtered through human attention and memory.
Affordance 2: Behavior pattern extraction
AI surfaced coordination patterns that were functionally invisible to human observers, including the finding that certain responder teams consistently concentrated near initial patient contact points while missing individuals in other areas of the building — a failure mode with serious operational consequences that no one had been able to see clearly enough to name. This connects directly to situation awareness theory (Endsley, 1995), which describes awareness not simply as perception of the environment but as comprehension of its meaning and projection of its future states. AI-enabled pattern extraction supported the move from perception to comprehension — the shift that makes reflection possible and behavior change likely.
As our 2025 Cambridge University Press chapter on promoting team situation awareness in fire and emergency response observes: “Introduced technologies will only be able to enhance TSA if they do not increase the burden of information processing or cause distractions” (Dailey, Dubrow, Bannan, & Santago, 2025). Every dashboard design decision was governed by that constraint, and every interface choice was tested against it.
Affordance 3: Situated replay for debrief
Research consistently demonstrates that most learning in simulation training occurs not during the event itself but during post-event reflection, with structured debriefs providing the primary mechanism through which behavioral data is translated into behavior change (Jenvald & Morin, 2004). Traditional debriefs, however, are limited by the same working memory constraints that limit observation: participants and instructors recall selectively, reconstruct events through shared assumptions, and rarely have access to objective behavioral evidence anchored to specific moments. CitizenHelper-training made it possible to replay targeted behavioral episodes during the debrief, timestamped, visualized, and grounded in sensor and video data, so that instructors could anchor the reflection conversation in what actually happened rather than in what people remembered or believed had happened.
Affordance 4: Hierarchical visualization reducing cognitive load
The dashboard applied split attention and redundancy principles from cognitive load theory directly to interface design (Sweller; Ayers & Cierniak, 2012), using progressive disclosure — broad situational overview first, granular event detail available on demand — to reduce the cognitive burden of processing multiple simultaneous data streams. First responders and instructors consistently preferred to receive warnings of high-level situational significance before granular operational detail, a preference that aligns with both the cognitive load research and with Endsley’s (1995) three-level model of situation awareness: perception, then comprehension, then projection. The redundancy principle guided our use of multiple streams confirming the same behavioral construct — not to overwhelm, but to increase confidence in findings that would otherwise be difficult to trust when acted upon under pressure.
—
What First Responders Actually Learned Better
The most consequential finding from this work was not one we had designed for. We expected the system to improve tactical performance metrics, and it did, but the outcome that carried the deepest implications for learning design was this: first responders and their leadership identified significant value in joint agency training for building inter-agency trust — a form of learning that observational methods alone had never made fully visible.
Teams who had never trained together before developed shared situational understanding through the debrief process itself, working from the same visualized behavioral data rather than from divergent agency-specific recollections of the exercise. The 3D digital twin was particularly valued by agencies unfamiliar with the arena’s layout, giving all participants a common spatial frame for discussing what had happened and where. Occupancy and location sensor data revealed coordination gaps between agencies that no one had previously been able to articulate clearly, because no one had been able to see them clearly — and once named and evidenced, those gaps became productive ground for reflection and behavior change in subsequent runs.
The AI did not teach these teams to trust each other; it gave them shared evidence precise enough to support honest reflection, and that reflection produced both the trust and the learning.
—
Five Transferable Principles for L&D Designers
This research program, conducted in one of the most demanding learning environments that exists, generated design principles that transfer directly to AI-enabled learning experience design in corporate and organizational contexts.
1. Start with the learning construct, not the tool. Before selecting any AI system, define what learning looks like behaviorally in your specific context. What does competence look like when demonstrated? What would you need to observe to know it had occurred? These questions should precede any conversation about technology.
2. Design for the debrief, not only for the experience. Because most learning happens in structured reflection rather than during the event itself, AI’s most powerful affordance in training contexts may be its ability to make debriefing evidence-rich. Consider where the designed reflection point is in your learning architecture and what AI-enabled evidence could make visible there that is currently invisible.
3. Apply cognitive load principles to every AI interface. Technology that adds information burden does not enhance learning — it impedes it. Progressive disclosure, integrated displays, and the redundancy principle are not optional interface preferences; they are learning science requirements that apply as directly to AI dashboards and tools as to any other instructional medium.
4. Identify what AI makes possible that was previously impossible. The affordance worth designing around is not what AI can do faster or more cheaply, but what it makes genuinely possible for the first time — in this case, capturing behavioral data across 70 people simultaneously, extracting coordination patterns across agencies, and replaying specific moments with objective evidence during a debrief. Finding the equivalent affordance in your design context is the analytical work that no tool selection process can substitute for.
5. Build trust as a design condition, not an afterthought. Every stakeholder group in this research raised data security and privacy concerns before committing to the system, and adoption of AI-enabled learning systems in any organizational context will depend on the same trust infrastructure — transparency about what data is collected, how it is used, and who controls it. Designing for trust from the beginning is not a risk management exercise; it is a prerequisite for the human-machine partnership that adaptive instructional systems require.
—
Conclusion
What a decade of research in emergency response training taught me — and what I find myself returning to consistently as I think about AI-enabled learning experience design more broadly — is that the most important work happens before any technology is selected. It happens when designers commit to understanding how the people they are serving actually learn: how attention operates under load, how working memory constrains what can be processed in the moment, how reflection produces behavior change, how teams build shared understanding across organizational boundaries. AI makes remarkable things possible in learning contexts, but what those things are, and whether they connect to genuine learning, depends entirely on the quality of the design thinking that precedes them.
The CitizenHelper-training system did not replace human instructors or human judgment. It gave human instructors something they had never had: complete, objective, visualized behavioral evidence to anchor the conversations through which first responders actually learn. That is the AI affordance worth designing for — not capability for its own sake, but capability in the service of learning that otherwise could not happen.
—
The multiteam systems research stream built on prior work supported by NSF EAGER award (2016–2018 PI: B. Bannan & N. Peixoto), Engineering Networked Devices for Multiteam Learning and Performance. The active violence incident simulation and CitizenHelper-training deployment were supported by the Center for Innovative Technology (CIT) SCITI Labs contract (2019–2020). The CitizenHelper-training system was developed in part under NSF award IIS-1815459 (PI: H. Purohit). Multiple companies and organizations donated equipment and participated in these events to make them possible including multiple First Responder agencies and most notably, Mutualink, Inc who donated the valuable time and expertise to instrument the building. We are most appreciative of all of these professionals involvement to make this research possible.
—
References
Bannan, B., Dubrow, S., Dobbins, C., Zaccaro, S., Purohit, H., & Rana, M. (2019). Toward wearable devices for multiteam systems learning. In Buchem et al. (Eds.), Perspectives on wearable enhanced learning (WELL) (pp. 79–95). Springer. https://doi.org/10.1007/978-3-319-64301-4_4
Bannan, B., Torres, E.M., Purohit, H., Pandey, R., & Cockroft, J.L. (2020). Sensor-based adaptive instructional systems in live simulation training. Adaptive Instructional Systems: HCII 2020 (pp. 3–14). Springer. https://doi.org/10.1007/978-3-030-50788-6_1
Dailey, S., Dubrow, S., Bannan, B., & Santago, A. (2025). Promoting team situation awareness in fire and emergency response. In T. Behrend (Ed.), Human-technology partnerships at work (pp. 143–155). Cambridge University Press. https://doi.org/10.1017/9781009348157
Dubrow, S., & Bannan, B. (2019). Toward improving situation awareness and team coordination in emergency response with sensor and video data streams. Learning and Collaboration Technologies: HCII 2019 (pp. 257–269). Springer. https://doi.org/10.1007/978-3-030-21817-1_20
Endsley, M.R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32–64.
Gallagher, P., Bannan, B., & Lewis, B. (2017). Next-generation learning: Smart medical team training. In Geng (Ed.), Internet of things and data analytics handbook (pp. 95–105). John Wiley and Sons. https://doi.org/10.1002/9781119173601.ch5
Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and development. Prentice Hall.
Pandey, R., Purohit, H., & Bannan, B. (2020). CitizenHelper-training: AI-infused system for multimodal analytics to assist training exercise debriefs. Proceedings of the 17th ISCRAM Conference. http://idl.iscram.org/files/rahulpandey/2020/
Purohit, H., Dubrow, S., & Bannan, B. (2019). Designing a multimodal analytics system to improve emergency response training. Learning and Collaboration Technologies: HCII 2019 (pp. 89–100). Springer. https://doi.org/10.1007/978-3-030-21814-0_8
About this article: The ideas, frameworks, and conclusions in this piece draw on Dr. Bannan’s own research and practice. Claude Sonnet was used to assist with drafting and editing. All content was reviewed and approved by the author.