Classical Meets Media
In Classical Music Meets Interactive Media, The main goal of this project is linking classical music and visual art through a real-time processing on a violin performance.
A meaningful 3D visualization will be correspondence within the original thinking of the composer after an exhaustive analysis of the score for the concert attending to the different parameters of the music language: harmony, counterpoint, instrumentation, formal analysis, style and background of the composer. The violin performer’s expression will be interacting with the music and the 3D visualization to create an interactive visual art in real-time.
The audience will be able to listen to classical music and at the same time appreciating the meaning of the musical scenes in visual art. The dynamic visual art projected behind the orchestra adds a new dimension to classical music appreciation and it helps the audience to apprehend the intention of the composer and the expression of the performer.
Description
As one of the most common aims of fine Arts and specially in the traditional thinking of Music, one of the earliest uses was in the accompaniment of theatrical dance or story-telling, so it is very usual that composers should from time to time produce what we know as “program music” – music written to portray events, activities or moods such as pastoral scenes or storms.
Thinking on an specific score, what defines a composer to another are a lot of little details that, along the time have been elaborated and transforming in one or other way; so there are some factor in a Mozart score that could never been attributed to Haydn. Those several items are taken carefully by the analysis process in order to distinguish the real parameters for a composer.
Pre-Concert Artistic Process
As an application tool for designing in real time there are different types of artistic parameters we have to consider previously.
Before designing what kind of image we are going to obtain by processing the music as an input signal we have to consider the artistic parameters are embodied as an inherent part of the score, and witch is the most important part of the process.
Musical Analysis of the score – Items:
Violin Bow Tracking: In-Concert Live Visual Art Creation
We will put two color strips on the violin bow (Figure 1) to track the bowing actions. Expressions like staccato, accent., pizz., can be captured by tracking the bow actions as well as other violin issues: Sautille, alla punta, tallone, sul tasto, sul ponticello, col legno, etc…
Real-time Sound Processing
Dynamics of the music piece like cresc., decresc., p, f, sfp, etc can be tracked using the microphone output and be represented into visual effects.
We will also track and process the pitch and beat of the violin sound in real time. Pitchddle~ patch is designed by Carlos D. Perales on Max MSP, based on fiddle patch design by M. Puckette and R. Climent. This pitch estimator and sinusoidal peak finder the pitch and amplitude of an incoming sound, both continuously and as a stream of discrete “note” events. Fiddle~ optionally outputs a list of detected sinusoidal peaks used to make the pitch determination.
The analysis hob size is half the window size so in the example shown here, one analysis is done every 512 samples (1.6 msec at 44.1kHz), and the analysis uses the most recent 1024 samples (23.2 msec at 44.1kHz). The minimum frequency that pitchddle~ with report is 2-1/2 cycles per analysis window, or about 108 Hz. (just below MIDI 45.)
By analyzing the pitch and beats of the live violin sound, we can compare it with the bow tracking to accurately determine the musical expression of the performer. The expression of the performer will in turn create and recreate the visual art displayed to the audience.
Visual Art
As a mathematical resource here are used Fractals: An irregular geometric object with an infinite nesting of structure at all scales. This is a geometric pattern that is repeated at ever smaller scales to produce irregular shapes and surfaces that cannot be represented by classical geometry. Fractals are used especially in computer modeling of irregular patterns and structures in nature. These shapes will be the base of the graphic design.
The term comes from “fractus,” which is Latin for broken or fragmented. It was coined by IBM Fellow and doctor of mathematics Benoit Mandelbrot, who expanded on ideas from earlier mathematicians and discovered similarities in chaotic and random events and shapes. We can see a moving fractal changing it own parameters:
Since the Fractal music is a result of a recursive process where an algorithm is applied multiple times to process its previous output we take the algorithm and connect it with the music analysis parameters. In wider perspective all musical forms, both in micro and macro level can be modeled with this process. Fractals provide extremely interesting graphic results, and the field is becoming one of the most exciting fields of new media research.
For instance, f(x) = f(x-1)^2 + c. This function makes the famous Mandelbrot set, named after fractal pioneer Benoit Mandelbrot. Here it is.
As we can see, in many cases, 200 iterations are needed to assign only one point. On most PC’s, a common number of points for a fractal is 303,200. This is why computers are needed to calculate the huge amount of iterations and to be precise.
The media artist who is handling this media performance will have control over some dynamics of the playing for example changes in the musical scenes and he can control the output of the visual effects. The actual visuals will be a combination of scenes, characters, and mathematical functions producing visual effects.
Project by:
Carlos D. Perales, Eng Tat Khoo, Joe Chung, Roger Thomas Tan, Corey Lee