Title | VIDEO LYRICS |
Brand | EAT KANAZAWA |
Product / Service | PRODUCT |
Category | G02. Branded Tech Offline |
Entrant | DENTSU INC. Tokyo, JAPAN |
Idea Creation | DENTSU INC. Tokyo, JAPAN |
Idea Creation 2 | QOSMO Tokyo, JAPAN |
Production | DENTSU CREATIVE X Tokyo, JAPAN |
Production 2 | QOSMO Tokyo, JAPAN |
Name | Company | Position |
---|---|---|
Kaoru Sugano | Dentsu Lab Tokyo | Creative Director |
Nao Tokui | Qosmo, Inc. | Executive |
Kouki Yamada | Freelance | Natural language processing programmer |
Kazuyoshi Ochi | Dentsu Lab Tokyo | Film Planner/ Copy Writer |
Ryosuke Sone | Dentsu Lab Tokyo | Film Planner/Director |
Jun Kato | Dentsu Lab Tokyo | Creative Producer |
Masafumi Fujioka | Dentsu Creative X. inc, | Producer |
Hiromi Nakamura | Dentsu Creative X. inc, | Production Manager |
Kiyonori Higuchi | Office Higuchi | Music Producer |
Ken Katsuno | Startline. inc, | Cameraman |
Mayumi Matsuzawa | Freelance | Hair & Make Up |
Tetsuro Isshi | Freelance | Animator |
We created an automatic lyric generation system based on the original karaoke video by employing image analysis via a convolutional neural network, enabling the perfect synchronization of the video and the lyrics.
A convolutional neural network learns from numerous combinations of photos and their descriptions. We then conduct a video scene analysis. A programmed AI creates captions for an unknown karaoke video frame. It repeats this process to quickly write more captions for the video. The system extracts distinctive words from the captions. It searches for related words and synonyms, then selects words for the lyrics. The system calculates the number of characters in the original lyrics, then replaces them with entirely new lyrics it has generated.
On January 29, 2016, we conducted a demonstration at the art event eAT KANAZAWA 2016. There, we presented new lyrics for existing Japanese songs in real time. As our official release, we presented a demo of original songs and video on the project website. At present, implementation as a new service and promotional activities are under consideration within the karaoke industry.
Karaoke videos were originally created as loosely-organized stories that would serve as a common base upon which to create wide variety of interpretations. By “showing” these videos to a computer and creating new lyrics that call to mind completely new stories, we aimed to give karaoke an entertaining new twist. And by perfectly syncing the lyrics and the visuals, we wanted to create a more immersive experience that would maximize people’s enjoyment of karaoke.