AR Spatial Informatics System

by Caleb Cram

Get Code  
Challenge
XR

ARSIS (Augmented Reality Space Informatics System) is a comprehensive software system conceptualized by the Boise State NASA SUITS team (of which I am the current team lead) as part of the NASA SUITS Challenge. Currently astronauts on extravehicular activity (EVA) procedures use voice communication and written instructions to guide them when completing tasks.

The Augmented Reality Space Informatics System (ARSIS) is an augmented reality (AR) prototype built to improve the efficiency of EVA protocols. The user is equipped with a Hololens 2 that can provide dynamic procedures that can be easily activated and traversed via Eye-tracking, voice control and physical hand tracking interactions. The Hololens 2 portal includes a geological sampling tool that allows the user to quickly and easily capture an image of the sample being collected as well as recording critical information about that sample and saving its location on a spatial mesh generated by the Hololens at runtime. This spatial mesh can then be sent to the desktop portal and the VR portal allowing other users to interact with the space being captured in real-time by the Hololens. The desktop portal has the ability to send/receive both audio/video messages along with shapes, arrows, text and lines drawn with the mouse. All of these will appear in the Hololens 2's viewport overlaying real world space allowing the desktop and VR portals to experience what the Hololens user is seeing and respond with dynamic, situation, spatial advice and directions. The VR portal has similar functionality to the Desktop portal but allows for a closer, more detailed view of what the user in the Hololens is experiencing. These three systems work together remotely to provide a seamless communication experience that can help NASA approach challenging, unknown environments with dynamic software solutions that adapt to the environment. To test this software, the desktop and VR portals must first be started, then the Hololens 2 can be started and will begin transferring meshes over the network to build out the space in the other two software portals.

For more:

TOOLS / STACK

Unity3D, MRTK 2, C#, HTML, JSON