Researchers develop a new way to instruct dance in Virtual Reality

Researchers at Aalto University were looking for better ways to instruct dance choreography in virtual reality. The new WAVE technique they developed will be presented in May at the CHI conference, a major venue for human-computer interaction research. 

Previous techniques have largely relied on pre-rehearsal and simplification. 

‘In virtual reality, it is difficult to visualise and communicate how a dancer should move. The human body is so multi-dimensional, and it is difficult to take in rich data in real time,’ says Professor Perttu Hämäläinen.

The researchers started by experimenting with visualisation techniques familiar from previous dance games. But after several prototypes and stages, they decided to try out the audience wave, familiar from sporting events, to guide the dance.

‘The wave-like movement of the model dancers allows you to see in advance what kind of movement is coming next. And you don't have to rehearse the movement beforehand,’ says PhD researcher Markus Laattala.

In general, one cannot follow a new choreography in real time because of the delay in human perceptual motor control. The WAVE technique developed by the researchers, on the other hand, is based on anticipating future movement, such as a turn.

‘No one had figured out how to guide a continuous, fluid movement like contemporary dance. In the choreography we implemented, making a wave is communication, a kind of micro-canon in which the model dancers follow the same choreography with a split-second delay,’ says Hämäläinen.

From tai chi to exaggerated movements

A total of 36 people took part in the one-minute dance test, comparing the new WAVE visualization to a traditional virtual version in which there was only one model dancer to follow. The differences between the techniques were clear.

‘This implementation is at least suitable for slow-paced dance styles. The dancer can just jump in and start dancing without having to learn anything beforehand. However, in faster movements, the visuals can get confusing, and further research and development is needed to adapt and test the approach with more dance styles’ says Hämäläinen.

In addition to virtual dance games, the new technique may be applicable to music videos, karaoke, and tai chi.

‘It would be optimal for the user if they could decide how to position the model dancers in a way that suits them. And if the idea were taken further, several dancers could send each other moves in social virtual reality. It could become a whole new way of dancing together’, says Laattala.

‘Current mainstream VR devices only track the movement of the headset and handheld controllers. On the other hand, machine learning data can sometimes be used to infer how the legs move,’ says Hämäläinen.

‘But in dance, inference is more difficult because the movements are stranger than, for example, walking,’ adds Laattala.

On the other hand, if you have a mirror in the real dance space, you can follow the movement of your feet using machine vision. The dancer's view could be modified using a virtual mirror. 

‘A dancer's virtual performance can be improved by exaggeration, for example by increasing flexibility, height of the jumps, or hip movement. This can make them feel that they are more skilled than they are, which research shows has a positive impact on physical activity motivation,’ says Hämäläinen.

The virtual dance game has been developed using the Magics infrastructure's motion capture kit, where the model dancer is dressed in a costume with sensors. These have been used to record the dance animation.

The WAVE dance game can be downloaded for Meta Quest 2 and 3 VR devices here.  The Github repository  also includes the open source code that anyone can use to develop the game further.

Reference:

Laattala, M., Piitulainen, R., Ady, N., Tamariz, M., & Hämäläinen, P. (2024). Anticipatory Movement Visualization for VR Dancing. ACM SIGCHI Annual Conference on Human Factors in Computing Systems. 

PDF of the study available here