Three-dimensional reconstruction and modeling technology is not new to hospitals. In fact, many institutions have radiologic technologists who specialize in performing 3D reconstruction from cross-sectional images.
PROBLEM
The results of their work are typically saved to a PACS, which is essentially a video with pre-defined angles – often referred to as a cine rotoscopio or cine turntable. While these “videos” are accessible to surgeons, the views and structures assigned create obstacles to fully understanding the patient’s anatomy.
“If surgeons could leverage the entire PACS suite, they would be able to create their own views; however, there is a significant mismatch between radiology-focused software users and surgical or interventional users,” explains Justin Ryan, director and research scientist at the Webster Foundation 3D Innovation Lab at Rady Children’s Hospital in San Diego.
Ryan, also a visiting professor in the department of neurosurgery at the University of California at San Diego, has a doctorate in biomedical engineering.
“Simply put, surgeons are often not trained to use the full range of radiology software to make the best use of the available 3D data,” he added.
This challenge, he continued, can best be illustrated through a recent case.
“A cross-sectional CT scan was performed for a patient with double outlet right ventricular anatomy and an abnormal airway,” he recalled. “As per our institution’s standard of care, the CT scan was 3D reconstructed in post-processing software. Two 3D reconstructed scans were created: one rotational and one tumbling.
“The surgeon provided feedback that they wanted the same rotation/inversion as the airway; then another request came in to make the aorta partially transparent, followed by additional transparency requests and unique angles,” he continued. “Our team created over 12 cines in an effort to meet the surgeon’s requests – where a comprehensive solution would allow the surgeon to perform all of these procedures in real time.”
PROPOSE
Ryan looked to other interactive fields, especially video game development, and found that there was a focus on creating an application that was easy to use (user interface) but effective (user experience). This theory is called UIUX.
“Our team sought to leverage the real-time 3D viewing capabilities present in the video game development space as well as UIUX principles to create a solution that would be used by surgeons and interventionalists as the primary users, rather than radiologists,” he notes. “We were fortunate to have a team member who had previously worked for a video game company. We were involved in this brainstorming process between surgeons, interventionalists, and innovation engineers to decide to use the Unity platform for real-time anatomy visualization.
“Our team sought to leverage existing real-time 3D viewing technology from the video game development space as well as UIUX principles to create a solution that is intended for surgeons and interventionalists as the primary users, not radiologists.”
Justin Ryan, Rady Children’s Hospital
“In working with stakeholders, we realized the main goal was to visualize 3D structures with as few obstacles as possible,” he continued. “To realize this goal, our team implemented periodic focus groups and pilot tests. We would give a simple prompt like ‘open the heart model lying on the desktop, then hide the aorta and make the airway partially transparent’ without any additional instructions.”
The team will monitor users and take notes at each stumble. Perhaps the icon isn’t ideal, or the implementation method – while making sense to the engineers on the team – ends up not making sense to the intended user.
“These hurdles would send our development team back to the drawing board,” Ryan said. “This continuous feedback from the intended user group during development was a significant benefit to our process. While medical software companies might use a medical advisory board, we had the advantage of continuous feedback from our team members.”
The result is Arc 3D Model Viewer, a 3D viewing platform specifically designed for reviewing 3D reconstructions in medicine.
Facing the Challenge
Ryan and the team eventually started with the Unity platform for real-time visualization. Many people may be familiar with popular video games like Pokémon Go, Marvel Snap, and Among Us. These games were built on Unity.
“Readers may not know that Unity also has non-gaming applications in the entertainment, education, and defense industries, as these industries also require real-time visualization,” Ryan notes. “The Unity platform allows us to create a solution where surgeons can fully interact with 3D datasets.
“We designed Arc 3D Model Viewer to fill the gap of real-time 3D viewing, not segmentation and reconstruction,” he continues. “Our process is to leverage FDA-cleared segmentation software – we use Materialise’s Mimics, but there are other platforms – to do the 3D segmentation and reconstruction. We export a final 3D model that can then be imported into Arc 3D Model.”
During development, staff continuously worked with the information security team as the ultimate goal was to deploy the software on hospital computers.
“Once we completed the release candidate application, we provided our information security team with the code to review for vulnerabilities; eventually, a third-party information security firm was also hired to provide additional review,” Ryan said. “With our approval of the software, it can then be deployed to hospital computers.
“Our users can now hide and reveal structures or virtually slice through structures in real time; this would not be possible with existing cines in PACS,” he added. “To further simplify the process, our engineers are building in DICOM data retrieval and query capabilities.”
RESULT
Ryan said the team’s biggest achievement was moving from idea to implementation within the organization.
“Going into a clinic and seeing Arc 3D Model Viewer on a computer being used to explain a patient’s condition to their family was personally rewarding,” he says. “We are now expanding our software branches to include patient education and training. After implementing our solution, we saw an increase in 3D requests, which I see as a testament to the promise of the software’s UIUX.
“We’ve been doing 3D reconstructions at the point of care for years, but the volume increased dramatically after implementing our solution,” he notes. “When we released the software for non-diagnostic use to the public, it was encouraging to see a spike in downloads, but a spike in a way we didn’t anticipate.”
Nearly half of the downloads came from academic or medical institutions outside the United States, which Ryan said was encouraging to see the potential uses for 3D beyond Rady Children’s patients.
FIND FREE TOOLS
“We have an academic and engineering mindset,” Ryan concludes. “We encourage people to reach out to us if they’re having trouble getting started or have ideas for how to visualize 3D in novel ways. We want to build collaborations that give more people more control over their 3D data.”
To download Arc 3D Model Viewer, click here. To contact Ryan and his team, email [email protected].
Follow Bill’s HIT coverage on LinkedIn: Bill Siwicki
Email him: [email protected]
Healthcare IT News is a publication of HIMSS Media.
