Mélodrame synthétisé par apprentissage audio-visuel (extrait)
Jérôme Grivel & Cédric Févotte © 2015

The video is an extract from the work of art "Melodrama synthesised by audio-visual machine learning" by French visual artist & musician Jérôme Grivel and data scientist Cédric Févotte (CNRS). They wished to question the idea of mathematical synesthesia and therefore of the relationship between physiognomical and artificial senses.

For that purpose, they have produced a computer-generated movie driven by music, based on the audio-visual analysis of a collection of movie excerpts. More precisely, they have developed a computer program that can identify and extract audio-visual patterns from the movies, in a so-called training stage. In a second stage, the patterns can be used to generate a visual stream from a given arbitrary audio input.

The extraction of the audio-visual patterns is performed by joint processing the pixels of the video frames with their time-aligned audio short-term spectra, using the mathematical technique of source separation, a field in which Cédric Févotte is a renown expert.

In the presented work, the audio-visual patterns were extracted from a collection of about 20 black & white movie excerpts staging a sentimental separation between onscreen characters. This very specific choice of movie scenes was made as a hint to Cédric Févotte scientific activity on source separation, itself at the core of the audio-visual synthesis system.

Besides, a tearful musical soundtrack was composed by Jérôme Grivel and used as an input to the visual synthesis system to produce a new movie with melodramatic value, thanks to the type of movies and soundtrack used respectively for training and synthesis.

The video is part of an installation consisting of analog TV screens displaying the movie excerpts used for training and a larger screen displaying the computer-generated melodrama (with musical soundtrack). The installation was presented at University Nice Sophia Antipolis (2014), Espace de l’Art Concret (Mouans-Sartoux, 2015) and at the Instants Vidéo festival (Friche Belle de Mai, Marseille, 2017).

Other resources
  • Catalog of the exhibition Looking for Search at University Nice Sophia Antipolis, where the installation was first presented (in French).
  • Pictures of the installation at Espace de l'Art Concret and black & white prints.
  • Scientific paper describing the audio-guided video synthesis system and presentation slides (in French).