Latency is one of the most negative experiences for users of immersive technologies like virtual reality, when using head mounted displays. The edge computing of the TERMINET platform will be a key component allowing participation in a training session, when distributed virtual teams are training over networks. The simulators are specific to a surgical intervention and its corresponding operative method or a medical treatment and its method. The simulators allow nurses, doctors and students to train at the same time technical skills, like bone drilling and cognitive skills like communication. They mirror a complete surgery or a medical treatment, giving each team member the option to train their own tasks.
The concept of Edge Computing, which enables low latency by increasing network efficiency with accelerated data processing, brings another advantage to group training in surgery. The staff members in the virtual reality session can participate in training without the head mounted display (HMD) and instead use a web browser for mobile or PC. The training application will be installed at the Edge node, streaming with full framerate the actual training to mobiles and PCs, using the Web RTC protocol. The non-VR users will be able to interact with the VR session also using web RTC.
The surgery training will be supported both by VR controllers, enabling haptic feedback, and hand interaction without controllers. A vital feature for the hand interaction is gesture recognition. The solution will enhance the default gesture recognition from Oculus Meta, by training the software further, using a neutral network. The objective is to better recognise gestures specific to medical use and to better predict the intention of the user. The federated learning approach of the TERMINET architecture will help the use case solution to extract the result of the local training at each VR device, update the algorithm at a central server and let it push back to the VR devices the new and aggregated result. There will be no exchange of sensible data, only the result of the local training, and less traffic in the network.