Kinect imaging lets surgeons keep their focus

New device lets surgeons consult medical images during an operation without having to scrub out

THE surgeon enters the operating theatre, covered in sterile blue scrubs. Machines beep and hiss. Nurses wait, tools at the ready: scalpel, forceps, bandage, Xbox... Xbox?

On Tuesday last week, a surgeon at Guy's and St Thomas' hospital in London began trials of a new device that uses an Xbox Kinect camera to sense body position. Just by waving his arms the surgeon can consult and sift through medical images, such as CT scans or real-time X-rays, while in the middle of an operation.

Maintaining a sterile environment in the operating room is paramount, but scrubbing in and out to scroll through scan images mid-operation can be time-consuming and break a surgeon's concentration or sense of flow. Depending on the type of surgery, a surgeon will stop and consult medical images anywhere from once an hour to every few minutes. To avoid leaving the table, many surgeons rely on assistants to manipulate the computer for them, a distracting and sometimes frustrating process.

"Up until now, I'd been calling out across the room to one of our technical assistants, asking them to manipulate the image, rotate one way, rotate the other, pan up, pan down, zoom in, zoom out," says Tom Carrell, a consultant vascular surgeon at Guy's and St Thomas', who led the operation on 8 May to repair an aneurism in a patient's aorta. With the Kinect, he says, "I had very intuitive control".

Carrell used the system to look at a 3D model of a section of the abdominal aorta, captured on a CT scan. This was projected on to a 2D live image-feed of the operation site, taken with a fluoroscopic X-ray camera. So Carrell could see what was happening inside the patient, as well as using the 3D model to help navigate the twists, turns and branches of the aorta. He says he consulted the system four or five times during the 90-minute operation.

Being able to check the images easily also helps surgeons maintain their concentration throughout the procedure. "You're just doing all of this stuff non-verbally and it just happens much more quickly. You're maintaining the flow of what's going on," says Carrell.

But manipulating a "touchless" medical image-viewer in a theatre filled with surgeons, nurses, machines, trays, cables and lights poses challenges of its own.

"You usually think of Kinect in a game-like scenario where you can jump around and move your hands as wide as possible, but surgeons are not allowed to reach such a large area," says Gerardo Gonzalez of Microsoft Research in Cambridge, UK, who helped develop the system in conjunction with surgeons from Guy's and St Thomas' and King's College London.

So Gonzalez and colleagues developed a set of gestures that a surgeon can perform in a constrained space, while standing at the operating table. For the most common actions - rotating the 3D model or placing a marker on the image - the team designed one-handed gestures that combine with voice commands, leaving the other hand free for operating. To position a marker, for example, the surgeon simply points at the image to activate a cursor and says, "place marker". Other functions, such as panning or zooming, require two hands.

Despite initial misgivings, Carrell is eager to continue working with the system. "I thought this was going to be a lot more awkward to start off with, but I was very pleased with the way it went today."

Free your mind

Manipulating medical images directly with gestures and voice commands doesn't just leave the surgeon free to concentrate on the task at hand, says Calvin Law, chief of surgery at Sunnybrook hospital in Toronto, Canada. It also frees up their memory. Law has completed a study of how surgeons used his Kinect-based medical-image viewer.
Surgeons usually have to look at a screen and memorise the images they need to work on. Law found that surgeons referred to images more often than normal and therefore felt fresher during surgery as they no longer had to hold everything in their own memory. His device is being developed by start-up firm GestSure, based in Seattle.