New increasingly accurate and fast technologies can track the movements of the human body. Combining motion data with sound provides tools for modern dance expression or augmented reality games, for example. These possibilities are being explored in the SonicMove project coordinated by VTT.
Information about the movements of the human body has proven useful in many applications. For example, in sports, movement patterns can be measured using a camera or wearable sensors. The collected movement data can be used for analysing the similarity or correctness of the repetitions.
VTT has a long history of developing wearable sensors where the measurement of motion is typically based on accelerometers, angular velocity, and magnetic field sensors. These sensors can be used to easily determine the relative motion, translation and rotation of the sensor as well as its absolute orientation. Typical applications for wearable motion sensors are step count- and activity level assessment, sleep monitoring and activity type recognition. VTT's aim is to exploit the potential of printed electronics to produce flexible and compact solutions that are practically unobtrusive in use.
When the motion data is combined with audio feedback, a whole range of new application opportunities can be discovered. Real-time audio feedback can be, for example, a continuous sound that changes pitch along with the speed of a hand. The sound can provide immediate feedback to the movement during a performance, for example in sports and gaming applications. Real-time sound can also increase the level of immersion in virtual reality interactions. The contact of your feet with a floor surface or the sweep of your hand on a table surface will generate a sound, depending on the surface properties. In virtual reality, this can be done digitally. Sound can be used to make stepping on a smooth floor sound like stepping on gravel or snow.
Motion-based sounds can also provide purely aesthetic experiences. When the sounds of different parts of the body are combined, they compose a soundscape that can be shaped by the movements – a work of art that combines motion and sound. However, if all the motion data is turned into sound, the result can be a messy cacophony. Therefore, it is essential that the data is properly pre-processed, and that only carefully selected parameters relevant to the final result are used for the actual sound synthesis.
Solutions used in a dance performance
The SonicMove project delves into how motion capture and real-time sound synthesis can be used in art and physical games. The first production of the project is a Biodata Sonata dance piece, which is being produced in collaboration with Dance Theatre Minimi. The work premièred on 15 November 2023 and will be performed six times during the autumn.
“The dance performance is an interesting example of how technology can be used in the arts. The arts and the gaming industry are potential applications for the technologies being developed by VTT. The measurement of human body movements and dance as an example of this are also closely related to health themes, which we study extensively at VTT,” says Teemu Ahmaniemi, Research Team Leader at VTT. The results of the project will be used, for example, in the rehabilitation of patients who have suffered a stroke or in the development of solutions to make everyday life easier for the visually impaired.
VTT's partners in the SonicMove project are the University of Eastern Finland and Aalto University. The project's corporate partners are Genelec Oy, a manufacturer of audio equipment, and Taustamarkkinat Oy, an advertising and marketing agency. Artistic partners include Kuopio Dance Festival, Dance Theatre Minimi and several dance and sound design professionals.
“On behalf of Dance Theatre Minimi and the artistic team of Biodata Sonata, we are very excited and enthusiastic about sonification as a method in the context of performing arts and about the collaboration with the Sonic Move project partners. The collaboration has already resulted in two new software interfaces, one for sensor data and one for sound design, as well as new artistic methods for choreographic, sound and visual design. At its best, sonification can deepen the sensory experience of art, and challenge artists and scientists to move towards something new. The collaboration between the dance theatre and the project partners has been open-minded, visionary, smooth and boldly blending the boundaries of the different fields," says Mikko Makkonen, Artistic Director of Dance Theatre Minimi.
The SonicMove project coordinated by VTT will last for 2½ years. The project is being funded by Business Finland for purpose of regenerating the creative industries.
Watch the trailer for a Biodata Sonata dance performance
Further information:
Teemu Ahmaniemi, Research Team Leader, VTT
[email protected], tel. +358 50 483 5425
Mikko Makkonen, Artistic and Production Director, Dance Theatre Minimi
[email protected], tel. +358 40 845 0754
