JukeBox music generation has been in development since 2019. It is still in an unreleased stage. After Sora became popular, it recently resumed R&D activity. We see many people under the official media channels discussing the issue of dubbing for Sora videos. It is highly likely related to Sora. Let us get to know Jukebox again.
Its original purpose was music generation. Its working principle is text-to-music: by editing lyrics, it generates the corresponding audio. If combined with Sora, I believe it should be capable of text-to-dubbing, because most people care about background music and lip-sync. Lip-sync in particular is a very complex task.
JukeBox Development Timeline
Selected Samples
Providing genre, artist, and lyrics as input, Jukebox outputs new music samples made from scratch. Below, we show some of our favorite examples.
Music Examples
Lyrics Conditioning
In addition to conditioning on artist and genre, we can also condition the model on the lyrics of a song to provide more context during training. A major challenge is the lack of well-aligned datasets: we only have song-level lyrics, not aligned with the music, so for a given audio chunk, we do not know exactly which part of the lyrics (if any) appears. We may also have song versions that do not match the lyric versions, which can happen if a given song is performed by several different artists in slightly different ways. In addition, singers often repeat phrases or otherwise alter lyrics in ways that are not always reflected in the written lyrics.
To match audio segments with their corresponding lyrics, we start with a simple heuristic that aligns lyric characters linearly across the duration of each song and, during training, passes a fixed-size character window centered on the current segment. Although this simple linear alignment strategy works surprisingly well, we found that it fails for some genres with fast lyrics (e.g., hip-hop). To address this, we use Spleeter32 to extract vocals from each song and run NUS AutoLyricsAlign [ ^reference-33] on the extracted vocals to obtain precise word-level alignment of the lyrics. We chose a sufficiently large window so that the actual lyrics have a high probability of being within the window.
To focus on the lyrics, we add an encoder to generate a representation of the lyrics and add attention layers that use queries from the music decoder to attend to the keys and values from the lyrics encoder. After training, the model learns more precise alignment.
The original information dates only up to February 2020. The recent restart was around March 1. We believe it is connected with Sora. Please follow us to read the latest reports.
|NIUBOYI
( 京ICP备17025393号 )|网站地图
