Project Summary
In 2022, a multi-institutional team of scientists and engineers completed a NOAA Ocean Exploration grant-funded project that focused on developing a sophisticated “co-exploration model” of human-robot interaction as a means of advancing and accelerating autonomous ocean exploration.
Planning autonomous underwater vehicle (AUV) surveys typically consists of finalizing a mission plan before each dive, which is then followed by the robot in an open-loop fashion. This means that a survey often requires multiple dives to the same site, using the results of one dive to plan for the next.
The “Ocean Co-Exploration Through Advanced Human-Robot Interaction” (CoEX) project team, on the other hand, developed a suite of tools that will enable scientists to adjust dive plans in real time based on acoustically transmitted information, enabling more efficient survey operations. The project’s goals were fourfold: (1) increase the pace of AUV-based ocean exploration, (2) reduce the reliance on crewed surface vessels and enhance the efficiency of using these crewed vessels when necessary, (3) demonstrate the use of a human-robot co-exploration model as a means of deploying advanced autonomous ocean exploration, and (4) generate a set of tools for co-exploration that can be broadly adopted by other groups running AUVs.
For the first goal of making AUV dives more efficient, the team made significant progress towards enabling real-time, data-driven redirection of AUV survey activities in the near term, and in the long term, incremental introduction and testing of subsea autonomy capable of making data-driven decisions independently. They developed and tested tools and a pipeline for single-dive execution of nested surveys over low-throughput acoustic links. During two expeditions in 2022, the team established an uplink of subsea-processed scalar data, including the visualization of those data on AUV Sentry's topside navigation and visualization display (February 2022), and an uplink of bathymetric maps and imagery from hybrid remotely operated vehicle (HROV) Nereid Under-Ice (NUI) (June 2022). CoEx-specific graphical user interfaces (GUIs) were used to display data products and query the subsea system for data in user-defined time/space windows. On the first expedition, they demonstrated a simple approach of autonomous subsea processing of chemical sensor data to identify what they called “hits” indicative of hydrothermal activities. These “hits” were communicated topside during a survey carried out by Sentry on the East Pacific Rise, a mid-ocean ridge along the floor of the Pacific Ocean. On the second expedition, CoEx tools allowed the scientists to demonstrate data-driven retasking of the vehicle during a single dive and multiple sensing modalities. They identified a likely wreck site in multibeam bathymetry, retasked the NUI vehicle to perform a survey at imaging height, and confirmed the presence of a wreck with uplinked images, all in real time during the dive.
For the second goal, to maximize the utility of low throughput communications links, particularly acoustic links and potentially satellite links, the team designed a system applicable to shipless operations, in which autonomous surface vehicles (ASVs) or other intermediaries work as relays between subsea vehicles and operators on shore. They demonstrated the CoEx pipeline between operators on Ocean Exploration Trust’s Exploration Vessel Nautilus and HROV NUI, with the uncrewed surface vehicle DriX acting as a relay during a NOAA Ocean Exploration Cooperative Institute (OECI)-supported expedition in fall 2021.
For the third goal, the team showed that it is possible for Sentry to use automatic substrate characterization to produce benthic maps. They applied an algorithm named “SUNSHINE” to previous imagery data collected by Sentry to perform autonomous classification of habitats and autonomous selection of representative images for human review. To this end, they converted a Sentry dataset and replayed the data as they were collected by the robot. The scientists then set up a pipeline that took as input Sentry imagery, localization information, and altitude measurements and ran the SUNSHINE algorithm for terrain characterization. The scientists obtained good results after adjusting the images and they also wrote a program for visualizing the produced streaming results (see video below). Their future work includes running SUNSHINE in real time on Sentry or another vehicle.
The team wrote the program above to visualize results. Video courtesy of Michael V. Jakuba. Download largest version (mp4, 91.7 MB).
With respect to the fourth goal, all CoEx applications and GUIs are open-source, made freely available for the public to access, modify, and distribute the code. Their availability was announced at the September 2022 Institute of Electrical and Electronics Engineers Oceanic Engineering Society AUV Symposium. These applications consist of proven and portable tools for visualizing data, multi-resolution bathymetry and photographic images, and selective querying for enhanced detail. Efforts are underway to integrate the CoEx technology with existing topside software infrastructure of the National Deep Submergence Facility at Woods Hole Oceanographic Institution . Building on and expanding the CoEx capabilities will support the deployment of sophisticated subsea autonomy and speed up the development of deep-sea autonomous ocean exploration, ongoing work that is being supported by the NOAA Ocean Exploration Cooperative Institute.
Contributed by Liang Wu, NOAA Ocean Exploration
Published November 9, 2022
Funding for this project was provided by NOAA Ocean Exploration via its Ocean Exploration Fiscal Year 2017 Funding Opportunity. Follow-on work is being supported by the NOAA Ocean Exploration Cooperative Institute (OECI).