Mixed Reality Lab- Classical Meets Media

Via Mixed Reality Lab

The Mixed Reality Lab (MXR) aims to push the boundaries of research into interactive new media technologies through the combination of technology, art, and creativity.

In Classical Music Meets Interactive Media, The main goal of this project is linking classical music and visual art through a real-time processing on a violin performance.

A meaningful 3D visualization will be correspondence within the original thinking of the composer after an exhaustive analysis of the score for the concert attending to the different parameters of the music language: harmony, counterpoint, instrumentation, formal analysis, style and background of the composer. The violin performer’s expression will be interacting with the music and the 3D visualization to create an interactive visual art in real-time.

The audience will be able to listen to classical music and at the same time appreciating the meaning of the musical scenes in visual art. The dynamic visual art projected behind the orchestra adds a new dimension to classical music appreciation and it helps the audience to apprehend the intention of the composer and the expression of the performer.

Description

As one of the most common aims of fine Arts and specially in the traditional thinking of Music, one of the earliest uses was in the accompaniment of theatrical dance or story-telling, so it is very usual that composers should from time to time produce what we know as “program music” – music written to portray events, activities or moods such as pastoral scenes or storms.

Thinking on an specific score, what defines a composer to another are a lot of little details that, along the time have been elaborated and transforming in one or other way; so there are some factor in a Mozart score that could never been attributed to Haydn. Those several items are taken carefully by the analysis process in order to distinguish the real parameters for a composer.

Pre-Concert Artistic Process

As an application tool for designing in real time there are different types of artistic parameters we have to consider previously.

Before designing what kind of image we are going to obtain by processing the music as an input signal we have to consider the artistic parameters are embodied as an inherent part of the score, and witch is the most important part of the process.

Violin Bow Tracking:

In-Concert Live Visual Art Creation
We will put two color strips on the violin bow (Figure 1) to track the bowing actions. Expressions like staccato, accent., pizz., can be captured by tracking the bow actions as well as other violin issues: Sautille, alla punta, tallone, sul tasto, sul ponticello, col legno, etc…

Real-time Sound Processing

Dynamics of the music piece like cresc., decresc., p, f, sfp, etc can be tracked using the microphone output and be represented into visual effects.

We will also track and process the pitch and beat of the violin sound in real time. Pitchddle~ patch is designed by Carlos D. Perales on Max MSP, based on fiddle patch design by M. Puckette and R. Climent. This pitch estimator and sinusoidal peak finder the pitch and amplitude of an incoming sound, both continuously and as a stream of discrete “note” events. Fiddle~ optionally outputs a list of detected sinusoidal peaks used to make the pitch determination.

See more!