EYE PLAY THE PIANO

Innovation Spike
TitleEYE PLAY THE PIANO
BrandTHE UNIVERSITY OF TSUKUBA’S SPECIAL NEEDS SCHOOLS
Product / ServiceTHE UNIVERSITY OF TSUKUBA’S SPECIAL NEEDS SCHOOLS
CategoryA01. Innovative Technology (incl. standalone solutions not in association with a brand or creative campaign)
EntrantHAKUHODO KETTLE Tokyo, JAPAN
Entrant Company HAKUHODO KETTLE Tokyo, JAPAN
Advertising Agency HAKUHODO KETTLE Tokyo, JAPAN
Production Company AID-DCC Osaka, JAPAN
Production Company 2 INVISIBLE DESIGN LAB Fukuoka, JAPAN
Production Company 3 TOKYO Tokyo, JAPAN

Credits

Name Company Position
Shota Hatanaka Hakuhodo Kettle Creative Director
Keitaro Kamijo Hakuhodo Producer
Masashi Ohashi AID-DCC Inc. Technical director/Programmer
Kenjiro Matsuo Invisible Designs Lab. Technical director/Sound producer
Yuri Morimoto AID-DCC Inc. Designer
Takayuki Kitai AID-DCC Inc. Motion designer
Yoshiaki Kajiyashiki AID-DCC Inc. Programmer
Masaru Kinoshita AID-DCC Inc. System Engineer
Kentaro Muraishi Planner
Toshiyuki Takei Tokyo/Taiyokikaku Co./Ltd. Producer
Kazuma Kitada Tokyo/Taiyokikaku Co./Ltd. Director
Masayoshi Takayanagi Tokyo/Taiyokikaku Co./Ltd. Production Manager
Rintaro Kozasa Tokyo/Taiyokikaku Co./Ltd. Production Manager
Miwa Onishi Taiyokikaku Co./Ltd. Production Manager
Shingo Ikeura Camera
Keiichi Tomita Camera
Ryosuke Toyama Camera
Takami Yuasa Blue Bayou Sound
Keiichi Nakajima Editor
Yoshitaka Honda Editor
Kota Numajiri Piano Player
Takashi Uno PR planner
Ryo Muramatsu P-point PR planner

The Brief

In Japan there are over one million seven hundred thousand (1,700,000) people with physical disabilities and three hundred (300) schools for the physically disabled where children with physical disabilities attend. The University of Tsukuba’s Special Needs-Schools, who operate various schools across the country for the physically disabled, declares "expanding the possibilities of children" as their educational philosophy and aims to provide an educational environment where handicapped students can express their imagination freely. In particular, children who face obstacles due to physical disabilities will have various limitations with regards to their “individual expression.” Kota Numajiri is a 16-years-old student of this school who suffers from Spinal Muscular Atrophy, a disease that affects muscular development in the arms and legs. He has loved music since he was a little child, and his dream was to play piano by himself. We then focused our attention towards “Music” as the medium for individual expression. With the help from the technology team we developed the innovative technology that would make it possible for physically disabled children to express themselves through music. After a six month development period we came up with a universal piano system called “Eye Play the Piano,” which enables a user to play the piano without using one’s hands or arms, but only with their eyes. The “Eye Play the Piano” system can be taught to the user anywhere as long as there is the exclusive application, headset display and a piano. The concept model of the application was developed in December of 2014 and is scheduled to become an open source application in December of 2015.

Eye Play the Piano started as a concept model put together by The University of Tsukuba’s Special Needs-School and the technology team but currently The Ministry of Education is considering the introduction into Special Needs Education. If the application is approved, the open source application of Eye Play the Piano will be completed by December of 2015 and is scheduled to be included in the curriculum in Japanese Special Needs-Schools from April of 2016. In addition, the charity funding that was put into effect during the project has already collected over one million (1,000,000) yen from countries all across the world. These funds are expected be used to donate the tracking head mount display device to Special Needs-Schools and also fund the updating of the application.

People with physical handicaps who have inconveniences with their hands or feet have various limitations with regards to“individual expression.”This is especially the case when it comes to playing a musical instrument in which case proves challenging to the normal person. The “Eye Play the Piano” system that we developed is the world’s first innovative system that enables a user to perform music without using their hands, arms or feet, but only with their eyes. We began by utilizing an eye-tracking system to recognize the direction of eyes through the virtual reality headset. The system detects the user’s eye movements to select the keys, and the blinks to play the corresponding note. About the system: 1. The virtual reality headset (FOVE) perceives the users eye movement through the mounted eye tracking technology. 2. The preferred note can be triggered by looking at the panel within the interface, which has been developed to be played by sight. 3. The user blinks on the preferred panels within the interface to trigger the selected note, which is then conveyed to the connected piano via MIDI signal. 4. Furthermore, tilting down of the head plays the role of the piano pedal, which lengthens the selected note. The “Eye Play the Piano” system can be taught to the user anywhere as long as there is the exclusive application, headset display and a piano. The concept model of the application was developed in December of 2014 and is scheduled to become an open source application in December of 2015. *FOVE refers to the eye tracking head mount display device. FOVE is the technology support member of this project. http://fove-inc.com/