United States

Artificial intelligence (AI) and machine learning (ML) systems are data intensive—requiring large amounts of data in order to train ML algorithms. Godbole (2020) states that AI/ML-based models are data hungry and the accuracy of the model largely depends on the quality and quantity of the data. As a reference point, it took approximately 40 days including 29 million games to train DeepMind’s AlphaGo Zero system to learn to play the game of Go (Feldman, Dant, Massey, 2019). Now imagine an AI/ML weapons system algorithm that needs to factor in concepts such as war context (war games, cold war, peace time), friend or foe, ethics, and legality (Feldman, Dant, Massey, 2019). The enormous amount of time needed to gather data and model such complex AI/ML systems will only be feasible if the Department of the Navy (DoN) has a central library for data.
There is currently no standardization of data collection across the DoN and no centralized library for data collection to be employed by future DoN AI/ML programs. This Naval Postgraduate School student-led project is taking a systems engineering approach to analyze the data needs of DoN AI/ML programs in order to develop requirements and a conceptual design of a Central AI Library (CAIL) system to support future DoN digital readiness for AI/ML systems.
The creation of a centralized library to access data points across the DoN could initiate the cultivation of data in order to provide future AI/ML programs with the quantity of data needed for consumption. The organic, real world data sets that would reside in the central library would provide a solid foundation to support future AI/ML algorithms. The library would have to be extensible to handle legacy needs and adaptable to support new requirements. The library would have to be able to load and query data simultaneously without degrading performance with the capacity to securely share and discover shared data.
Data gathered at the lowest levels could be used to build higher level concepts and complex structures to support AI/ML systems. The application of these data for DoN AI/ML programs would include domains such as weapon systems, maintenance, training, and security. These domains have the same barrier to entry that exist in the DoN for AI/ML programs; accessing data points that have been aggregated over time. If these AI/ML programs could select the best-case scenarios generated from the centralized library, these programs would be able to proceed to real world testing validated from test scenarios that were correctly predicted based on real event testing. Establishing data requirements early allows for information to be collected for future use. Additionally, if the Navy isn’t careful with the data that feeds AI/ML, the output could be biased, skewed, or wrong. The data collection methods, standards, and integrity of the data are being explored as part of this NPS project. 


NAML 2021

Data Management for Artificial Intelligence and Machine Learning Implementation Across the Department of the Navy

data

data collection

artificial intelligence

machine learning

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

technical paper

Given the growth of adversary capabilities, learned algorithms need to be trained to offer tactically complex recommendations that can be statistically analyzed to ensure mission success, warfighter acceptance and naval superiority. This research investigated Event-Verb-Event (EVE) ontological constructs to address this training need. Two projects were involved: (1) Battle Readiness Engagement Management (BREM) decision aid (currently used at the Naval Postgraduate School for training algorithms) based on NWC documentation, and (2) an autonomous package delivery service modeled at the training data level. The EVE construct mapped complex mixes of tactical actions to statistical results for use as training data for supervised and reinforced algorithm applications. BREM research involved 21 EVE classes/categories with over a hundred attributes. Each class/category was modeled using defined attributes which were simulated for statistical analysis. The autonomous delivery service involved five classes and 17 attributes. From both projects, Naïve Bayes, Logistic Regression, Random Forest, XG Boost, k-Nearest Neighbor, Q-Learning (using reinforced learning) and a mini-max algorithm feeding a non-linear optimization routine were investigated. When describing the first or second event (E) within the EVE construct, an event is modeled as status from a sensor, communication link or human input. An EVE attributes/features and consumes no simulation time. Status can convey a measurement of mission success, e.g., package delivered within an upper and lower bound confidence interval of time. The verb (V) within the EVE construct represents the core of the action, consuming simulation time and always initiated by a predecessor (or preceding) event defined by its input attributes. Successor events represent a measurable mission outcome. If one or more predecessor events are reported, then an action will occur. If an action occurs, then one or more successor events are reported. A successor event becomes a predecessor event for the next EVE in the sequence of tactics. By using conditional probability dependencies to cause statistical independence between predecessor and successor events, meta-model outputs from simulations can train algorithms. Conditions for statistical independence are based on modeling status defined by attributes for an EVE/ML class. Attributes within the model represent the state or predecessor event. The meta-model output of the EVE constructed simulation, combined with the input attributes, completes the synthetic training set for supervised and reinforced learning algorithms. Because the meta-model output is a measurement or statistical analysis of measurements, the algorithm can learn statistical output, supporting user understanding of the odds of mission success from the ML result. The results of the research demonstrate how EVEs can model complex mixes of tactics represented as ML class, their related attributes and mission success statistics. It also showed that by defining tactical sequences/threads using EVE constructs, the developer can use simulation output to train algorithms that identify complex courses of action based on the highest likelihood of success.

Using Event-Verb-Event (EVE) Constructs to Train Algorithms to Recommend a Complex Mix of Tactical Actions that can be Statistically Analyzed

Naval Ordnance Safety and Security Activity (NOSSA) determined a need to ensure quality and quantity of training data is sufficient for systems deploying Artificial Intelligence (AI)/Machine Learning (ML) Algorithms. Acquiring quality and quantity of training sets becomes increasing difficulty when limited to synthetic data. Research in Generative Adversarial Networks (GANs) applications focuses on synthetic generation of images using neural networks. Inspired by this approach, this research describes the use of generative, discriminative constructs within a modeling and simulation environment. The generator would model and simulate sparse and missing data supporting “realistic” operational conditions – creating training artifacts. When the discriminator approves the artifacts, the algorithm is ready to be trained on synthetic situations involved with sensor malfunction, message corruption and human input errors. Results demonstrate effective training of a variety of ML algorithms to handle external and internal inputs, e.g., sensor tracks, communication links or human input, all non-image related artifacts. We developed an autonomous package delivery simulation to measure reliable delivery time. Using generative, discriminative constructs, Naïve Bayes, Logistic Regression, Random Forest, XG Boost, k-Nearest Neighbor were trained to support selection of autonomous systems and the most reliable route. The generative construct consisted of randomly generated input variables for the simulation, e.g., speed, distance, weather, surface quality. Availability of status sources were randomly selected to represent missing and sparse data issues that the ML algorithm might encounter during an operational deployment. For the discriminator, our research developed two measurement techniques. The Training Set Alignment Test determined the quality of the training set, i.e., representing operational realism, and the Simulation to Attribute Ratios for nth (StAR-n) Order verified if there was sufficient training data quantity, i.e., attribute variation modeled. Data sets were generated, i.e., modeled and simulated, to target a specific ML algorithm for training. StAR-n showed that the total number of artifacts did not determine sufficient quantity for algorithm training. Instead, quantity determination was based on the number of attribute types. TSAT ensured accurate mapping of requirements to attribute types, therefore supporting the quality of the model and related results for training. Both measurements showed that quality and quantity were tightly coupled to meet operational deployment needs. If the quality and quantity of the entire training data did not meet acceptance criteria defined in both measurements, then the entire training set was determined as fake. The training data was then discarded and a new training set was generated for discriminator review using the two measurements. We were able to minimize generative, discriminative cycle times by shaping the random generator selecting attributes for model simulation to more accurately replicate deployment issues involving status availability to the algorithm, e.g., sensor malfunctions, communication link failures and human input error. The generative, discriminative constructs using simulates and measurements ensure that the quality and quantity of data sets can train ML algorithms to adequately perform in operational conditions when status availability is uncertain.

Applying Generative Adversarial Network constructs to Mission-based Simulations to produce “Realistic” Synthetic Training Data for Machine Learning Algorithms

Casey Sapp with VRTUL (www.VRTUL.co) will discuss his ROV client’s recent successes using VRTUL’s AR/VR teleoperations platform to reduce the burden on operators. By utilizing a combination of Unreal Engine and AR/VR headsets VRTUL is rapidly enabling ROV pilots to perform complex tasks at full ocean depth, and concurrently building photorealistic synthetic simulations to better train ROVs to perform autonomous tasks on their own. Leverett Bezanson with Seebyte will highlight some of the software architecture involved for high speed RL, simulation, and an AI/ML roadmap to fully unsupervised ROV autonomy.

VRTUL and SeeByte have partnered on two SBIRs for the USN dealing with AI and ML as it applies to maritime robotics command and control. Specifically, those SBIRs are “N192-099 3D Visualization Capability for Fleet Remotely Operated Vehicles (ROVs)” and “N202-133 Multimodal Interaction Technologies to Support Small Unit Leaders”. This talk will discuss the transition of these two efforts to the fleet and how they complement each other through the use of AI and ML. 

Casey Sapp is CEO of VRTUL (Virtual Reality Technology Underwater Limited) focused on underwater AR/VR technology solutions. VRTUL's clients include the Navy, BBC, OceanX, National Geographic, Monterey Bay Aquarium Research Institute and many others in need of high performing underwater multi-camera arrays.

Leverett Bezanson is the Engineering manager for the SeeByte US office. Leverett Leads all of the U.S. based engineering efforts. SeeByte is involved in many ongoing Office of Naval Research, JIEDDO, and Naval Sea and Air Systems Command projects, as well as the US Navy EOD MK 18 Family of Systems Program of Record. SeeByte also has commercial presence in the offshore oil and gas industry and has achieved some significant milestones and firsts in autonomous platform operations, pipeline, and cable tracking. The US office is responsible for much of the development, testing and integration of cutting-edge Autonomy, AI, and ML software for the maritime industry.

ROV VR and High Speed Reinforcement Learning

The Georgia Tech Research Institute has been supporting the JAIC Joint Logistics virtual sensing National Mission Inititive called VSTAR. Through the development of the VSTAR digital twin MBSE and engagement with DoD transition partners, the development team has assembled a discussion on AI/ML transition best practices. The presentation will give a brief VSTAR background for context, why a digital twin virtual sensor was developed for the HH-60G, and the critical lessons learned from development through transition to successfully deliver a new capability to the warfighter.

Requirements Considerations for AI/ML Digital Twin Model Transition

The outbreak of COVID-19 on the USS Theodore Roosevelt in March of 2020 presented the Navy with a challenge: manage a novel disease outbreak with limited resources. In response, Defense Digital Service deployed “My Symptoms”, a system in which sailors reported a variety of symptoms to assess their need for immediate medical intervention. The Cloud and Analysis Support during COVID-19 Response (CASCR) group at the Naval Information Warfare Center (NIWC) Pacific supported the deployment of “My Symptoms” for multiple sites while collecting, visualizing, and analyzing the data; this lead to actionable insights for the care of quarantined sailors. The CASCR group later applied machine learning techniques to the symptom data to and showed that data collected for triage purposes could be used to determine if a sailor was a symptomatic carrier of SARS-CoV-2.

Operational and Investigative Uses of Symptom Tracker Data from USS ROOSEVELT and USS KIDD

This session presents the results of a project to leverage the significant investment that commercial industry is making in AI that is very focused on improving commercial shipping outcomes, and see if similar models and algorithms can be applied to the DoD space. 
This project was planned and executed under the direction and funding of NS2 Labs (https://www.ns2labs.com/). NS2 Labs is a research organization focused on combining capabilities from the government, academia, commercial industry in order to provide solutions for the US DoD and Intelligence Community. 
We selected three use cases to evaluate on this project: Anomaly Detection, Destination Prediction, and Zone Activity Prediction. The hypothesis behind this project was that commercial AI capabilities could be successfully applied to these DoD maritime use cases. We were confident in this hypothesis largely because of scale: as a company, SAP manages over 70% of worldwide logistics data. The resulting large number of commercial customers means that SAP invests significant amounts of capital to optimize the analysis of this logistics data; presumably at least some of the resulting models would be relevant to the DoD maritime domain. Our research identified five primary model sets used extensively by SAP’s commercial customers: 
1. Port Characteristic Modelling - these models account for water level, wind, waves, visibility, vehicle characteristics and congestion in the waters within and surrounding a port. 
2. Automated Maritime Geofencing – these models support geofencing of specialized areas, detection of concealed vessel identity, brushing incidents, pollutant shipping analysis, and correlations between vessels/businesses/individuals. 
3. Port Operations Modelling – these models address the cargo details, vessel details, visitors/staffing, vehicle operations, security operations, and financial/operational performance of the overall facility. 
4. Demand Forecasting for Transport – these models assess more than 1700 worldwide macro-economic factors to predict future activities across the logistics spectrum. 
5. Global Track and Trace – these models assess the entire logistics and delivery process for SAP’s commercial customers to allow shippers to get in front of potential issues. 
Project execution consisted of processing AIS and radar data with the SAP/Intel Corporation data management platform HANA, enhancing data and model environment with subset of the identified commercial capabilities, and training models to determine home/destination ports, normal behavior by region, anomalous behavior, activity expectations by region, and destination prediction with continuously updated confidence values. 
We continue to work through the remaining commercial components. Our next area of research is bringing in more non-traditional data sources based upon mobile device tracking, supply chain interconnectivity, and social media content overlays. 

Adapting Commercial AI Approaches to Support Maritime Domain Awareness

The Digital Aviation Readiness Technology Engine (DARTE) provides unprecedented predictive readiness capabilities for the Naval FA-18 fleet. DARTE focuses on discovering actionable insights in relation to predicting two key readiness metrics: the number of mission capable (MC) aircraft and flight hours. Recent DARTE improvements are particularly noteworthy, including the adoption of cutting edge AI and deep learning techniques such as temporal pattern attention mechanism enhanced long short-term memory (LSTMA) networks, hyper-deep ensembles for enhanced performance, and improved uncertainty estimation and robustness. Hyper-deep ensembles and attention mechanisms have been shown to provide state-of-the art results in industry and academia. Furthermore, their improved uncertainty estimation provides decision makers with an increased level of confidence that allows for better, smarter decisions.

Improving The Digital Aviation Readiness Technology Engine with Temporal Pattern Attention Mechanisms and Hyper-Deep Ensembles

The United States Navy’s stated force-structure goal is a 500-ship fleet by 2045, a significant build-up from today’s 293 ships. The expanded surface fleet will be designed to meet this goal with operational capabilities, and within operational environments, that have evolved since the last surge of naval shipbuilding in the 1980s. These new challenges will likely drive increased diversity in the design characteristics of future surface combatants—a diversity requiring tools able to rapidly inform set-based designs with high-confidence predictions. This research provides such a tool. Utilizing machine learning, the residual resistance coefficients of a new hull form, across a desired range of speed-to-length ratios, are accurately predicted. With parametric, nondimensional hull form coefficients and ratios from historically tested ship hull methodical series, parametric, non-parametric, and ensemble method algorithms were trained, validated, and tested. The research analyzes the necessity of first classifying a hull design to an established methodical series prior to performing regression efforts to predict resistance. Conversely, the research also analyzes the benefit of machine learning regression algorithms to predict resistance from an entire hull design data set, without first classifying. Finally, the research demonstrates the power of machine learning in quickly adapting to small amounts of additional training data to accurately refine the predictions. The intersection of machine learning and hydrodynamics is a little-explored frontier that, even in its infancy, provides dynamic resistance predictions that are more accurate than Holtrop’s statistical method with a speed dependent form factor. Through the establishment of a shared hull form geometry and performance repository, the true power of machine learning may be unleashed, removing the most challenging of obstacles associated with preliminary hull form design.

Machine Learning in the Prediction of Displacement Hull Form Resistance

The understanding of large data sets and developing predictive capabilities is imperative for the understanding of any boosted system.
Currently, sensor data and historical data is underused due to constraints on data processing capabilities. Data from well-sensored static tests or flights can be vast in size, requiring extensive computing capabilities and processing time.
Using machine learning techniques and high processing computers can provide data processing techniques that are the most useful for the user and maximize the results from the data available and provide the most insight on future tests and the effects of physical changes to the hardware. These tools combine to develop a predictive reliability framework for missile-centric propulsion systems. The machine learning approach uses combinations of sensor data, image processing techniques, and acoustic analysis along with the subject matter expertise of aerospace and propulsion engineers to develop a useful data processing system that can be adapted to many applications. This work describes a novel process for the analysis of experimental plume data sets using multimodal machine learning techniques and analytics in order to develop virtual sensors, detect anomalies, and predict performance based on historical data.

This approach will supplement physically acquired data during ground static testing of solid rocket motors, as well as increase performance measurement above the limited numbered of digital flight instrumentation. Data acquisition for the virtual sensing effort is performed using a solid booster static test stand developed at GTRI. In order to capture the plume characterization with physical sensors, the test instrumentation suite includes axial thrust, on-board acceleration and casing temperature and external imaging, temperature, and acoustics measurements. The machine learning approach focuses on developing anomaly detection, shock detection, and sensor reconstruction solutions through virtual sensing and image processing algorithms for predictive performance analysis of a solid booster. Results show the pre-processing of experimental data, including the resulting dynamics during a test fire, and the ability to recreate different sensor signals based on their inherent behaviors. Additionally, the anomaly detection approach gives autonomous detection of off-normal events and different methods of edge detection were used to identify shocks within the plume.

Virtual-Sensing for Predictive Reliability

If a feature in a dataset can be identified as a potential causal factor through an objective process, this data can then be used to feed any number of conceptual approaches to prevention of safety mishaps. One of the many ways in which this analysis can be attempted is via longitudinal analysis of specific traits attributed to personnel, either individually or in aggregate. That is to say, by examining specific traits as they change over time for their predictive value with respect to a specific outcome. Each moment that a variable is measured results in a feature that is highly correlated with the feature derived from the same variable in the previous or next moment. This presents unique challenges for fitting regression models since fitting highly correlated features will muddle the attribution of causal factors and result in a reduced ability to distinguish the relative importance of features. Additional challenges with regard to analyzing safety mishaps are that the data is frequently imbalanced, meaning that there are very few mishaps when compared to the number of instances in which a mishap did not occur, and that it is more important to be correct about the potential causal factors extracted than it is to be quick. This paper will walk through the process of comprehensively sampling the feature space, applying a progressive framework of winnowing tests designed to reduce the number of models considered to a computationally manageable number while removing irrelevant features, and then developing the results into a workable set of models. Case studies for this process using specific examples are available in the appendices.

Downloads

Next from NAML 2021

Using Event-Verb-Event (EVE) Constructs to Train Algorithms to Recommend a Complex Mix of Tactical Actions that can be Statistically Analyzed

Similar lecture

Improving the Efficiency of the Naval FA-18 Fleet Using The Predictive Cost of Readiness Engine (PCORE)

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from NAML 2021

Using Event-Verb-Event (EVE) Constructs to Train Algorithms to Recommend a Complex Mix of Tactical Actions that can be Statistically Analyzed

Similar lecture

Improving the Efficiency of the Naval FA-18 Fleet Using The Predictive Cost of Readiness Engine (PCORE)

Downloads