Subcommittee Category: Human Performance Analysis and Engineering

Abstract: Measuring mission-critical trust between human operators and collaborative synthetic teammates is a priority within the DoD to achieve third-offset goals, accelerate automation design and training for hybrid human-machine teams, and support next generation multi-domain warfare. Achieving proper trust calibration has long been a primary mechanism by which Human-Machine Team (HMT) performance can be maximized by avoiding system distrust and over-trust. However, proper trust calibration hinges on the implementation of effective trust calibration techniques based on real-time trust assessment. The current study establishes the relationship between HMT trust, workload, and performance in a Search and Rescue (SAR) paradigm where human operators supervise intelligent Unmanned Air Vehicle (UAV) assets to achieve mission success in a constructive synthetic environment. A novel trust measure was developed and piloted in this experiment to precisely measure subjective trust variations across time and in conjunction with target task elements. Thirty participants, including UAV operators and novices, participated in a rigorously controlled, within-subjects experiment that involved supervisory control of intelligent UAVs promoting collaborative decision-making via system recommendations across four SAR missions. Workload was manipulated by alternating the number of UAVs to supervise across each trial. Trust was assessed via our novel measure in addition to established metrics. HMT mission outcomes were measured via Measures of Performance (MOPs) and Measures of Effectiveness (MOEs). Objective biometric-based metrics were also used to measure operator workload using the Cognitive Workload Classifier (CWC), and Index of Cognitive Activity (ICA). Statistical analyses describe the relationship between trust, workload, and performance, and the impact of automated recommendation accuracy on HMT trust and mission outcomes. This experiment is unique as it provides a foundation for a real-time self-report measure of trust that can be directly compared to real-time physiological measures. Study findings further discuss intervention techniques to maintain proper trust calibration in operational environments.

Subcommittee Category: Simulation

Abstract: Advances in artificial intelligence (AI) such as natural language processing (NLP) and reinforcement learning (RL) are enabling a resurgence in immersive flight training for both instructor-led and self-paced training, such as with immersive MR/VR (Mixed Reality/Virtual Reality) devices.

Introduction of MR/VR will likely require dialog management and command & control capability in a self-paced learning context. Adding an NLP-based cognitive agent acting as a virtual instructor and co-pilot provides the required immersion level to broaden the spectrum of self-paced learning during the pilot’s learning journey. Pilots receive instant feedback on their performance, explanations of the communication procedures, and progress tracking as they develop their skills.

Accent-tolerant advances in speech interaction are used to recognize radio transmissions from the ownship using NLP on a flight training knowledge base. Conversation agents increase student immersion and offer more realistic workloads by fully automating the ATC (air traffic control) function, freeing up the instructor to focus more on core observation and training tasks, and allowing more automated flying without an instructor for MR/VR simulation training.

The complexity of real-world ATC communications, such as conditional clearances and instructions that include give-way, can only be simulated when the ownship is fully embedded with other traffic. An AI ATC module leverages a collaborative multi-agent framework to manage air traffic during real-time MR/VR flight simulation in a synthetic urban environment scenario.

This paper will explore issues around robustness in reinforcement learning and evolutionary optimization problems, alongside new results in collaborative multi-agent systems. These outputs will provide further results in nonlinear function approximation (e.g. deep neural networks) and optimization methods in stochastic environments. It will also study human factors during immersive Mixed-Reality training of emergency scenarios with an AI agent integrated into an eVTOL (electric Vertical Takeoff and Landing aircraft) flight training simulation and operation platform.

Subcommittee Category: Human Performance Analysis and Engineering

Abstract: An automatic pilot assessment capability using machine learning algorithms that can inform a flight instructor during a flight training session in full flight simulators is proposed in this paper. The current research explores a hybrid expert system and machine learning capability to assess pilot performance in flight simulation. Hybrid rule-based and machine learning algorithms are considered in the approach. Assessing a pilot’s performance during a flight training session is a capability that can considerably improve the effectiveness of a training session and help the flight instructor provide better instructions and feedback. In this paper, we investigate an efficient way to build an automatic objective assessment engine, that provides a performance index that uses both knowledge of subject matter experts and instructors to train the artificial intelligence capability. By using multi-labels that have the same meaning but come from different sources of knowledge, we demonstrate that an automatic assessment engine is able to reduce the subjectivity of the instructor and optimize the time of the rules creation, tuning, and testing effort for the expert system development. In addition, we show that this hybrid approach increases the accuracy and precision of the assessment of pilot maneuvers during training sessions by using a consensus methodology that blends the multiple sources of knowledge.

Subcommittee Category: Emerging Concepts and Innovative Technologies

Abstract: The Joint Live Virtual Constructive (JLVC) federation has evolved for the past 20+ years to meet mission needs for Joint Training. The 2022 National Defense Strategy calls for a revolutionary approach to integrate data and software to speed their delivery to the warfighter. There is a need to rapidly provide realistic experimentation, rehearsal, and training to the Joint Force. In addition, DoD directives for data analytics and implementation of a zero trust architecture (ZTA) are forcing modernization across the Joint Training Enterprise. The JLVC is modernizing to enable faster integration of Service, agency, and partner simulations. While modernization is necessary, it should build on the established governance processes and standards that have enabled the existing JLVC to support a steadily increasing number of joint and service training events.

The vast amounts of data that are currently required to support the design, planning, and execution of events should align with authoritative data sources, and enable analytics to inform concept development. To maintain an operational advantage over adversaries, the DoD has defined a data strategy based on treating data as a strategic resource central to all warfighting levels. Valuable data generated during a Joint Event Lifecycle should be processed into usable information and made available efficiently and at scale.

This paper presents a conceptual architecture that incorporates modern data mesh principles while building on the existing foundations of the JLVC federation. This approach uses the concepts of unity of effort and treating data as a product to provide the necessary level of coordination and cooperation to work towards common objectives. Adding a self-service infrastructure with the consistency provided by an ontology provides the agility and flexibility to quickly enable the use of the authoritative data while supporting future data analytics at both speed and scale.

Subcommittee Category: Training

Abstract: Developing and validating innovative solutions to train student pilots as effectively as experienced Instructor Pilots (IP) is a priority for many defense and civilian aviator training programs around the world to increase student throughput, minimize impact of IP shortages, and reduce overall training costs. Innovative training paradigms target the development of low-footprint, immersive simulators that maximize training task coverage and training effectiveness in self-paced environments when aptly paired with digital training solutions that can mimic the behaviors and evaluation heuristics of expert IPs. The current study investigated the utility of training using a Virtual Reality (VR) simulation-based training device paired with a next-generation synthetic IP providing real-time coaching, feedback, and scoring along with immersive and gamified debrief capabilities aimed to maintain student motivation and engagement. Thirty cadets from a large Indo-Pacific Asian Air Force participated in an hour-long training event practicing basic maneuvers across time. Difficulty was manipulated by alternating time of day. Maneuver performance was assessed automatically against syllabus-based criteria. Cadet workload was assessed via both NASA-TLX and the biometric-based objective Cognitive Workload Classifier (CWC). Pre and post surveys were administered to gauge cadets’ confidence and overall training system perceptions. Results show significant training effects across time, along with a decrease in cognitive workload trend. Results are further discussed in terms of cadets’ perceptions by experience and performance levels. A key finding shows a strong motivational effect for cadets when using the training system with synthetic coaching and feedback. This study is unique as it was field tested in a germane operational pilot training environment and proves the viability of core aspects of next generation training solutions. Study findings are discussed to address overall strengths and limitations of next-generation pilot training solutions, as well as important consideration for integrating such system within existing and new training courses.

Subcommittee Category: Policy, Standards, Management, and Acquisition

Abstract: The core component of a simulation-based training system is the process of creating training scenarios. The scenario creation process is essential for simulation-based training systems since scenarios are designed to provide the context for the training to occur. An effective training scenario should provide opportunities for trainees to practice their skills, and receive feedback from the instructors on their performance. A training scenario is normally characterized by three main components: (1) the initialization, (2) the key events that must happen during the training, and (3) the termination conditions. Prior to developing a training scenario, training objectives must be analyzed to determine the set of knowledge and skills that are required as part of the training to ensure that scenario outputs are domain-valid and pedagogically effective. Because of this reason, the creation of validated and effective training scenarios must be carried out by qualified instructors and highly trained subject matter experts. The process is challenging and time-consuming, therefore expensive. The objective of this paper is to describe an approach that will facilitate the task of the generation of training scenarios. The proposed method is based on the ontology of knowledge presentation. The paper presents an ontology developed to capture simulation scenario attributes that are pertinent to the flight simulation domain, and describes the role of this ontological analysis in the process of creating training scenarios. Leveraging the SISO interoperability standard, this study expands the C2SIM core ontology by adding a Training Scenario Extension layer to it. As a result, training scenarios can be generated in a standardized format that complies with the base and extended C2SIM ontology. Finally, the paper will present a use case of air refueling training as an application of this approach. This work constitutes an important step towards standardizing practices in automated simulation scenario development for flight simulation applications.

Subcommittee Category: Simulation

Abstract: Wargames bring together the two concepts of simulation and games to offer structured and rigorous environments where players can explore strategies, concepts of operations, and technologies across different levels of war. Simulations ingest military doctrine and performance data into their high-fidelity platform models. Then, using a game construct, players enter plans that are carried out by high-fidelity models in which outcomes are compared against objectives. The future state-of-the-art U.S. Marine Corps wargaming facility at Quantico is capable of multiple simultaneous games across multiple classification levels. The simulation environment attempts to be as realistic as possible to provide an immersed training experience comparable to a real-world battle. For instance, weather represents an important factor in determining the course and outcome of battles. Rain can slow the movement of a force, or wind intensity can alter the range of a weapon system. Environmental data, such as terrain, wind, precipitation, turbulence, and other meteorological parameters are examples of the limits of the weather condition profile of a simulated environment. Converting these weather parameter features into quantitative effects and impacts is not only computationally burdening for simulation systems, but also compromises the “fair fight” aspect of the simulation since each simulation system computed the weather effect differently. Therefore, the weather data must be transformed into effects and impacts to be effective. The challenge for wargame designers is to provide accurate and timely weather data to the simulation systems, but also tactical decision aids that relate the impact of the weather on systems performance. This paper examines the role of weather simulation in the USMC Wargaming capability. It describes how the historical weather data augmented by a dynamic weather simulation model is used in wargaming. Finally, it will evaluate how the simulated weather is translated into effects and impacts on the simulation systems during a wargame’s execution.

Subcommittee Category: Emerging Concepts and Innovative Technologies

Abstract: To improve and accelerate pilot training, this paper explores the capture of cognitive/psychophysiological states using biometric sensors and flight telemetry to drive an intelligent human performance assessment system in immersive 3D flight simulation device.

Assessing pilot performance during a training session is a capability that can partially be performed by an AI-based algorithm. With technical data gathered during a flight maneuver, such assessment can provide objectivity during flight training, can be a predictor of future pilot performance, and adapt simulation training using a combination of flight telemetry (technical skills) and biometric/behavioral data (non-technical skills).

Evaluation of non-technical skills remains difficult without the support of data analytics and proper visualization tools. Additionally, soft skills are inherently more difficult to grade compared to technical performance. An AI engine can provide cues on behaviors and cognitive/psychophysiological states that cannot be easily observed by the instructor.

With a cohort of 16 novices, we explored neuroscience capabilities that could enable real-time adaptive flight training using a variety of data collected from a flight training session. By using electroencephalogram (EEG), eye tracking device and flight telemetry data with N-Back, BART & IGT cognitive baseline methodologies in a fast-jet flight simulator with an e-Series Medallion visual, we intend to provide a training scenario & maneuver analysis during initial training for both technical and non-technical flight performance.

Subcommittee Category: Simulation

Abstract: Next generation frameworks have been developed to support large-scale pattern-of-life simulations. These simulations are paramount to assess the consequence of hazardous events in urban environments and to develop effective emergency response strategies to these events. CAE has been prototyping large-scale pattern of life simulations in real time, human-in-the-loop operations for concept development, course of action analysis, and training. The simulation of bomb explosions is a critical component of emergency response simulations and provides valuable insights into the potential effects of an explosion on infrastructure and populations. This paper will report three simulation scenarios of Ammonium Nitrate/Fuel Oil (ANFO) explosions in urban areas of Tallin, Estonia. The blast simulations were simulated with Blastfoam, a compressible flow solver for high-explosive detonations and airblasts that integrates easily with other computational fluid dynamics (CFD) software such as OpenFoam. Detonations ranging from 5,000-10,000 lbs of a fertilizer lorry in the areas of Tallin’s Freedom Square, the Stadium and Town Square were simulated. The simulations considered the variability of explosive charge, surrounding infrastructure including buildings, terrain topology and street corridors. The framework was integrated within CAE’s core servers, including the importation of 3D urban geometry from CAE’s visual databases and CAE’s Single Synthetic Environment (SSE) server for visualization of the blast pressure waves. A 76-core Azure Virtual Machine was used for meshing and running the CFD solver for up to 1000 milliseconds. The blast pressures imported into the SSE server converted spatial and temporal distributions of pressure data into probabilities of human casualties. The paper will demonstrate the effectiveness of using advanced frameworks and pattern-of-life simulations as a crucial tool for training emergency responders and evaluating the efficacy of response plans.

Subcommittee Category: Simulation

Abstract: Gray-zone activities―behaviors and/or actions potentially leading to, but below the threshold of armed conflict― executed across actors’ instruments of national power present significant national security and global stability challenges. Successful gray-zone maneuver depends on an actor’s ability to model, then implement effective strategies whilst managing the associated risks and chaos of possibly destabilizing activities. One approach to modeling the evolving nature of global competition and conflict is examining the history of international relations encoded in multiple Conflict and Mediation Event Observations (CAMEO) open-source databases. An exemplar – the Global Database of Events, Language and Tone (GDELT) project is 55TB of events and related data from public news sources accumulated for four decades. This paper examines the feasibility and suitability of this data as a means for decision-makers to explore complex, dynamic gray-zone phenomena, anticipate competing incentives, and assess consequences of choices. There are four facets to this proposed approach. First, it will be shown gray-zone news events fall on a Pareto distribution in terms of the number of mentions each gets in the media. Second, Reflective Thematic Analysis (RTA) is used to extract relevant data from GDELT to train statistical topic models for actor behaviors. Thirdly, results―including newsfeeds and thematic signatures―are generated for two actors over the first four months of 2023. Regarding data quality it will be shown that filtering events with low mention counts can be used for data conditioning, but unfiltered and filtered topics appear statistically similar so that strong filtering is not usually worth the information loss. Finally, we discuss utilizing open-source intelligence (OSINT) for potential model generation for wargaming capabilities. In this, emphasis will be placed on the usefulness of mention counts for cost-benefit-risk analysis to aid decision-making as well as the power of RTA to adapt OSINT to alternate analyst frameworks