Project Overview

AI-powered pre-operative surgical planning and training virtual reality (VR) platform for image-guided therapy (IGT)

Motivation and Objective

Image-guided therapy is a collection of techniques to make surgeries less invasive and more precise, which improves recoveries and outcomes. Surgical planning is an important of the approaches undertaken by clinicians and medical imaging plays a central roles in this process. The types of imaging used as well as the tools used with this imaging are always involving with the field. However, most clinical centers use bespoke, locally developed, data processing pipelines and visualization tools. Therefore, these systems are difficult to extend or to share across centres. Also due to their bespoke nature, AI-integration is difficult.

The IGT-VR-AI project extends Luxsonic’s SieVRt VR medical image workflow and MLhub AI platform to create a complete IGT workflow powered by these technologies. The project is currently funded at Phase II of the INOVAIT grant funded by Canada’s Strategic Innovation Fund and includes collaboration with Sunnybrook Research Institute, the Memorial University of Newfoundland and Voronai Health Analytics.

This project is still being actively developed.

Key Contributions

  • Complete technical scoping across frontend and backend functionalities.
  • Time estimation of development activities.
  • Integration of existing AI inference algorithms.
  • Guidance of UI and UX workflow and components.
  • Led technical and scoping meetings with all skateholders.
  • Frontend development.
  • Backend development.

Technologies Used

The technologies that are/were used.

  • Frontend: Unity3D, C#.
  • Backend: Python, Flask, Bash, Docker.
  • Tools: Git, Jira.

Challenges and Solutions

  • Challenge The project is a greenfield area and was initially quite open-ended.

    • Solution I focussed on on key areas of clinical interest of our collaborators as well as well-developed inference models to identify three key use cases for the project.

  • Challenge One of the inference models provided by a collaborator was only available via a Docker container and was quite resource intensive.

    • Solution I extended MLHub to also encapsulate algorithms from standalone Docker containers and implemented resource management functionality which allowed the use of this inference model.

  • Challenge SieVRt’s existing diagnostic workflow differed from this project’s intended workflow, particularly with the way data is ingested.

    • Solution I worked with our team to identify legacy code and modules, refer to planned changes and collaborated to integrate our project’s desired functionality into these changes.

  • Challenge UI components that would trigger some functionality being actively developed did not yet exist during early stages of the development process of VR/frontend components.

    • Solution A clever system of hot-keys was mapped as a substitute for UI components to achieve the desired functionality instead.

Impact and Results

INOVAIT
SieVRt