Context-Aware Gesture Controller

Project Overview

A gesture-controller that works with any medical image viewer (DICOM viewer) to replace/augment the mouse and is context-aware based on eye-tracking.

Motivation and Objective

In the early stages of my startup journey at ClearVoxel, I would ask radiolgists open-ended questions about what they disliked about their work. Almost everyone stated they hated the computer mouse. Research showed that 70% of radiologists suffer from some type of overuse injury from mouse use. I have even heard, with a new tool being pitched at a conference, radiologists in the back of the room chanting “No more clicks!”. The mouse is used for multiple functions, but primarily for navigating the images during the diagnostic interpretation. This project sought to replace the computer mouse for image navigating tasks by offering a more ergonomic alternative.

Key Contributions

  • Conducted, recorded and analyzed user interviews.
  • Specified functionality of the proof of concet and support development.
  • Tested and demonstrated the proof of concept to several radiologists.
  • Improved the accuracy of gesture detection by moving from rules-based detection to a trained Support Vector Machine model.

Technologies Used

  • Frontend: C#, WPF, Tobbi SDK, LeapMotion SDK
  • Hardware: Tobii 4c eye-tracking, LeapMotion Controller

Challenges and Solutions

  • Challenge Building a medical image viewer from scratch is a huge undertaking, even if using an SDK and radiologists have their own preference.

    • Solution We built the proof of concept as an overlay so that it would work with any existing viewer running on the same machine.

  • Challenge The Leap Motion Controller is traditionally placed on the desk with the hand raised above it, which just created new ergonimic issues.

    • Solution I motified an adjustable phone mount to could mount the Leap Motion Controller to an office chair and place it in an area that was out of the way and would let the user perform their gesture actions with their arms comfortably on the arm rest.

  • Challenge The hand tracking of the gesture controller was not perfect and therefore sometimes gesture would not be recognized to do rigid, rule-based gesture detection.

    • Solution I implemented detection based on Support Vector Machines which was easy to train using the camerea fast frame rate. This had the added benefit if making the gesture detection tailored to the user.

Impact and Results

  • We turned the feedback from our interviews into this prototype in a matter of 3 weeks, leaving some of interviees in disbelief that we had actually built it.
  • Gesture detection was improved an estimated 3X using SVMs.
  • Ultimately, we had made a very expensive mouse replacement and we had concerns about the financial viability of this as a product.