Dance videos are ubiquitous on online video sharing sites. Therefore, there is an increasing demand for automatic processing content-based dance videos. A recent article on arXiv.org looks at the rhythm of a dance video, specifying the musical pitch amplitude as well as the dancer’s visual movement.
Image credit: arXiv: 2207.03190 [cs.SD]
The researchers suggest that the dance movement pattern (visual rhythm) should be in sync with the musical rhythm and propose to use this correspondence as a supervised signal for self-supervised learning. Recommends a joint music-dance performance and a dance rhythm separator that is convenient for music-dance comprehension and reproduction tasks.
Tests on tasks such as dance classification, music-dance retrieval, and dance music retargeting confirm the effectiveness and generalizability of the proposed model across musical scenarios -dance.
While audio-visual representation has proven applicable to many basic tasks, the representation of dance videos is more specific and always accompanied by music with complex auditory content. complex, still has many challenges and has not been paid attention to. Considering the intrinsic link between the dancer’s rhythmic movement and musical rhythm, we introduce MuDaR, a new Music-Dance Performance learning framework for performing rhythmic synchronization of music and dance in both explicit and implicit ways. Specifically, we infer dance rhythms based on visual imagery and movement cues inspired by musical rhythm analysis. The visual rhythms are then time aligned with the musical rhythms, which are extracted by the amplitude of the sound intensity. Meanwhile, we exploit the implicit association of rhythms implied in audio and visual streams by contrast learning. The model learns to embed matching by predicting temporal consistency between audio-visual pairs. Music-dance performance, together with audio and visual rhythm detection, can be further applied to the following three tasks: (a) dance classification, (b) music retrieval- dance and (c) retargeting music-dance. Extensive tests demonstrate that our proposed framework outperforms other self-monitoring methods by a large margin.
Research articles: Yu, J., Pu, J., Cheng, Y., Feng, R., and Shan, Y., “Self-supervised learning to perform music-dance through clear rhythmic synchronization- tacit”, 2022. Links: https://arxiv.org/abs/2207.03190
