Making music with EyeGaze in the real world

This blog post charts the continuation of my work with BBC Performing Arts Fund fellow Bram Harrison a.k.a DJ Eyetech. Bram describes himself as living with ‘locked in syndrome’ and we’re working together to enable him to make music on his EyeGaze computer. This music will then form part of the next installment of Bram’s series of Eye Life radio shows for Phonic FM.

The project so far has been fraught with problems, exposing the folly of assuming that new and innovative technologies such as EyeGaze will play nicely, despite the claims made by the manufacturers. EyeGaze has huge potential but requires very different functionality from software than when using a conventional keyboard and mouse/trackpad as the means of control. It can also be very temperamental.

Back in 2013 Bram took delivery of a brand new Tobii EyeGaze system to replace his ancient Tobii P10. But after months of trying and with input from all kinds of people it had to be concluded that the system simply didn’t function in a way that afforded Bram reliable control of his communication software – The Grid 2, let alone any kind of music software, whether ‘mainstream’ or adapted. The new computer was ditched and the old P10 resurrected. The situation is complicated by the fact that Bram is only in a position to use one eye to control his computer. But while this works reasonably well on the older P-10, for some reason it wasn’t good enough for the ‘new improved’ Tobii machine.

During this time I was researching existing music software that Bram might be able to use via EyeGaze, as documented on the Drake Music Eye Gaze Music Resources forum thread. (This forum also includes a piece about the challenges of EyeGaze where I use a ‘Bram Harrison approved’ analogy of  trying to hammer a nail into a wall one-handed…)

The ideal aim for Bram is to find music software that he can use independently, at his home in rural Devon, without the need for anyone to do anything other than position him in front of his computer and switch it on. It also needs to be software that is suitable for an adult (not something developed for a child!) that he can use to develop his music making over a sustained period of time. These requirements narrow the options considerably.

At the foot of this page is a short film showing Bram trying out two contrasting pieces of music software for the first time:

• The first is The EyeHarp created for EyeGaze users by Zacharias Vamvakousis as part of his Master’s Thesis. EyeHarp is a self-contained musical instrument consisting of a matrix sequencer on one page for creating looping melodic patterns, as well as a circular interface on another page for playing notes in real time and transposing the key of the loops playing from the matrix sequencer. All sounds are contained within, and can be selected from inside the software.
• The second is EyeKeys, created by Barry Farrimond of the MUSE Project as a simple, uncluttered interface for playing sounds in real time via MIDI using EyeGaze. In this example Bram’s computer is connected to my laptop via MIDI, playing a sound from Native Instruments Absynth

I’ve intentionally left each section of the film quite long to try and give a sense of how long things actually take in Bram’s situation.

In the film below it’s clear to see that while using Eye-Harp the cursor is constantly jumping around on the screen, even though Bram is working hard to keep his eye still. This makes it very difficult to ‘dwell’ the cursor over any object for long enough to ‘click’ on it. Although Bram does well to make some music with EyeHarp on this first attempt, the cluttered layout and the lack of options to modify either the layout or the dwell-time make this software difficult to use.

The layout of EyeKeys consists of six large rectangles with ‘rest areas’ in between, which, as the film demonstrates, is much more realistic for Bram to use for playing music. EyeKeys would require more development for Bram to be able to use it independently, starting with an option to select from a menu of built-in sounds. But simply for the purposes of testing the relative usability of the different screen layouts, the contrast between EyeHarp and EyeKeys was extremely useful, both for myself and for Bram to better understand what his needs and interests are.

Other options we considered testing included:

• EyeGaze Loop Composer by Adrian Gierakowski, which provides a very elegant way for EyeGaze users to play and mix pre-made loops in Ableton Live. But Bram would still be reliant on others with the necessary music tech skills to create Ableton set-ups for him to use. Plus, the software simply wouldn’t have functioned on Bram’s ancient Tobii P-10 anyway.
• A Brain-Computer interface, by Dr. Mick Grierson, which potentially avoids the problems associated with an EyeGaze system by enabling Bram to create music just by thinking about it. However, while this kind of technology sounds really exciting, the reality was that there was no way such a system could be set up and left for Bram to use independently at his house in rural Devon.

Finally, Bram also tested E-Scape – a multi-track MIDI Sequencer that has been built from scratch with accessibility in mind.  E-Scape has been accessible to switch-users for years, but is now being developed by Dr. Tim Anderson for EyeGaze users. Comparing E-Scape to EyeHarp and EyeKeys led Bram to conclude that he would prefer to focus on composing and sequencing music rather than playing in real time, partly because EyeGaze doesn’t give him the level of control he would need to express himself musically. However, E-Scape also has performance functionality that Bram could take advantage of later on. Bram is now beginning to use E-Scape to create music, and I’m aiming for this to be the subject of my next blog and short film.

Thanks to the BBC Performing Arts Fund for funding this project.

Video