Medical Training Comes to Life in VR Simulation
The 76-year-old patient in the doctor’s office suddenly becomes mute and can’t move her right arm—classic symptoms of a stroke.
Image courtesy of the University of Maryland Institute for Advanced Computer Studies.
Cheri Hendrix, director of the physician assistant (PA) program at the University of Maryland, Baltimore (UMB) goes into assessment mode, her voice growing urgent. “Tell me what day it is,” she says as she checks for a heart rate. “Tell me where you are.”
The scene isn’t playing out in real life. It’s a 3D training simulation representing a common stroke scenario. Soon, PA program students outfitted with virtual reality (VR) goggles will be able to immerse themselves in the experience, watching Hendrix evaluate the patient as if they were with her, seeing what Hendrix sees and even “walking around” in virtual space to view the exam from different vantages.
The imaging facility used to create the scene, called the HoloCamera, was built by University of Maryland, College Park (UMCP) computer visualization experts in response to studies showing that people retain more information in immersive training environments. The 3D medical training simulation project using the HoloCamera is a partnership between the UMCP and UMB campuses through the University of Maryland Institute for Health Computing.
“Sitting still and using a desktop and mouse doesn’t aid in forming memories as well as experiencing things in an embodied context,” said the project’s lead investigator Amitabh Varshney, a professor of computer science and dean of the College of Computer, Mathematical, and Natural Sciences at UMCP. “Immersive interactivity is more educationally engaging than watching a video or flipping through a book.”
Image courtesy of Amitabh Varshney
In September, Varshney’s team unveiled the technology to a pair of PA students. At the HoloCamera studio in College Park, second-year clinical student Ashlee Leshinski donned goggles and started moving her head from side to side. A wall monitor captured her fluid view as she observed Hendrix assessing the patient, a professional actor recruited for the project.
“Oh, wow, that’s super cool,” a grinning Leshinski said as she moved around the room and looked down, as if perched above the patient’s chair. “You can see her perspective and all the angles. It’s so clear, too.”
Later during an anatomy education demonstration, Leshinski and a fellow clinical student stepped into a second immersive environment. This one featured an animation of the human cardiovascular and neurological systems overlaid on the patient from head to toe. A UMCP team member guiding Leshinski through the simulation was able to trigger a visual representation of an irregular heartbeat that sent a blood clot to the patient’s brain and impaired nerve impulses down her right arm, connecting the outward symptom of immobilization with internal causes.
Cheri Hendrix (left) and Amitabh Varshney
The HoloCamera features volumetric capture technology, relying on 300 high-resolution cameras precision-mounted in a green room studio to film subjects in 360 degrees. The collective ultra-high-definition footage was fused together using UMCP-developed hardware and software featuring artificial intelligence.
“Medicine is one area where precision matters,” said Varshney. “That’s not possible with low resolution.”
The project was partially funded by a $1 million National Science Foundation grant led by Varshney. Development kicked off several months ago when Hendrix filmed the stroke scenario with the actor inside UMCP’s studio. The team worked with medical professionals to create the anatomical overlays.
The HoloCamera was developed amid an increased demand by clinical training facilities for instructional tools that feature lifelike environments. Varshney’s research has shown that people in immersive environments recall information almost 9% better than people who watch traditional instructional videos or read textbooks.
To study the HoloCamera’s effectiveness in a stroke patient setting, Hendrix later this fall will compare a group of students trained with the tool with a control group receiving traditional classroom instruction. In the years ahead, the team plans to apply the tool to anatomy education and surgical training at the University of Maryland Medical Center in Baltimore.
Not only will the technology help students in College Park and Baltimore, but it can also be scaled to assist practitioners in rural areas and far-flung countries, bringing the talents of Maryland professors and clinicians to populations in need, the developers say.
“This will be a boon to education all over,” Hendrix said. “Sky’s the limit.”
Written by John Tucker