R&D

VIDEO LYRICS

2016.08.16

case film is here

Background

Karaoke is a well-known form of entertainment from Japan which is loved all over the world. But unfortunately, even though the music and lyrics on the screen continue to change, the video playing in the background is often times not the song’s official music video but rather an automatically-generated compilation of outdated stock clips, preventing people from really enjoying themselves at karaoke and actually becoming a cause of stress. Karaoke-goers have long been puzzled by the illogical, nonsensical videos that accompany karaoke tracks.

our idea

We created an automatic lyric generation system based on the original karaoke video by employing image analysis via a convolutional neural network, enabling the perfect synchronization of the video and the lyrics.

Karaoke videos were originally created as loosely-organized stories that would serve as a common base upon which to create wide variety of interpretations. By “showing” these videos to a computer and creating new lyrics that call to mind completely new stories, we aimed to give karaoke an entertaining new twist. And by perfectly syncing the lyrics and the visuals, we wanted to create a more immersive experience that would maximize people’s enjoyment of karaoke.

step1

A convolutional neural network learns from numerous combinations of photos and their descriptions. We then conduct a video scene analysis.
0815_karaoke-02
0815_karaoke-03

step2

A programmed AI creates captions for an unknown karaoke video frame.
0815_karaoke-04
0815_karaoke-05

step3

It repeats this process to quickly write more captions for the video.
0815_karaoke-06

step4

The system extracts distinctive words from the captions.
It searches for related words and synonyms, then selects words for the lyrics. The system calculates the number of characters in the original lyrics, then replaces them with entirely new lyrics it has generated.
0815_karaoke-07

We conducted a demonstration at the art event eAT KANAZAWA 2016

There, we presented new lyrics for existing Japanese songs in real time.
As our official release, we have presented a demo of original songs and video on the project website.

At present, implementation as a new service and promotional activities are under consideration within the karaoke industry.

credit
Executive Nao Tokui(Qosmo)
Creative Director Kaoru Sugano(Dentsu Lab Tokyo)
Natural language processing programmer Kouki Yamada
Film Planner/ Copy Writer Kazuyoshi Ochi(Dentsu Lab Tokyo)
Film Planner/Director Ryosuke Sone(Dentsu Lab Tokyo)
Creative Producer Jun Kato(Dentsu Lab Tokyo)
Producer Masafumi Fujioka (Dentsu Creative X)
Production Manager Hiromi Nakamura (Dentsu Creative X)
Music Producer Kiyonori Higuchi (Office Higuchi)
Cameraman Ken Katsuno (Startline)
Hair & Make Up Mayumi Matsuzawa
Animator Tetsuro Isshi

Dentsu Lab Tokyo