User Tools

Site Tools


"VirtualPhilharmony": A Conducting System with Heuristics of Conducting an Orchestra

Authors Takashi Baba, Mitsuyo Hashida, and Haruhiro Katayose
Publication Info Retrieved here
Retrieval date 3/23/11

Proposes a way of interfacing conductors (of symphonies, etc) to computers.

Interfaces for conductors

Conducting interfaces can be thought of as a form of active listening. An effective system is usable and provides the user with the experience of conducting a real orchestra. They were first introduced in the 1980s - the first was the radio baton. Often the movement of hand is detected and beat/tempo information is deduced, which may be used to set the progression of a piece of music. Few systems take into account the fact that the musicians do not always completely obey the directions of the conductor. One system takes this into account by calculating the tempo based on the last four beats, the deviation of the current beat, and the expected tempo (in a fixed manner). VirtualPhilharmony (VP) attempts to tackle this problem with a “Concertmaster Function”, where a set of heuristics about the musicalities/intentions of the orchestra members are used to calculate the tempo.


VP detects the local minimum of hand movement using various detectors, and treats that time index as the beat point. The first sensor is a glove, equipped with an Arduino, accelerometer, and bluetooth, which is considerably lighter than a wiimote (an option used by other projects). The use of bluetooth means the orientation doesn't matter, and because it's confined to a glove the movements can be as wild as the conductor desires, but the accelerometer is orientation-dependent. The second method is a baton with an infrared LED, which is detected by a wii remote. Because of light interference and directionality, this baton is not very reliable. Finally, they use a capacitive sensor (MIDI theremin), which is highly accurate and precise but is limited in range (must be 20 cm away from antenna).

Coordination of conductor and music

VP uses a score-based template to inform its analysis, which includes score and deviation information (attack, release, volume, tempo deviations). This information is fed into the “concertmaster function”, which coordinates the conductor's movements with the template. This models how all musicians and the conductor have the same score, but change their playing depending on one another and their idea of the music and style. In order to ensure that the template “follows” the conductor, it is modified dynamically based on the conductor's actions. Changes in tempo are treated as continuous from beat-to-beat, approximated by linear interpolation (called the “integration tempo scheduler”). The score is also displayed to the user.

Concertmaster Function

The concertmaster function serves three purposes: Revising the template per-beat tempo, per-bar style, and per-beat rhythm pattern based on the conductor's movements; predicting subsequent beats based on beat times and musical style; and adjusting the lag between conducting and music. The per-beat tempo is revised based on a weighted average of the conductor's tempo and the template's tempo. In order to quantify the rhythmic pattern of each bar, the time taken by each beats is found and normalized to the bar length (to form ratios) - for example, for a waltz, the second beats are longer than the first and third beats. These ratios change with tempo, and this relationship is modeled by a linear least-squares. The rhythm pattern (amount of the bar used) of dotted notes is also changed on a per-beat level according to the tempo (also modeled by linear least-squares). In order to get an idea as to how to predict subsequent beats, the correlation between the next beat location and N previous beats is found, using linear predictive coding to minimize the error. Eight pieces with four rhythmic/tempo structures were analyzed (all beats in time, phrases beginning with auf-tact, extreme tempo changes, and waltz style). Depending on the resulting error for calculating the first, second, third, etc subsequent beat, the technique can be changes. From the examples above, for fixed tempo, N is fixed as the number of beats per measure; for auf-tact, N is even; for highly varying tempo N is also fixed as the number of beats per bar but if the tempo changes more than the sensory threshold, it's set to 1 temporarily; for waltz, each beat uses a different number of prior beats for prediction. The time-lag between conductor and music is set as a constant which depends on tempo and a weighting coefficient.


The system was received well by a professional conductor, who suggested that trying to make the tempo slower did not work well. Various other players suggested that the limited range of the infrared camera was problematic and suggested that a computer generated conductor could help teach new musicians how to conduct. As expected, relatively fixed-tempo pieces were well-modeled by the system, but some problems occurred with pieces whose tempo changed dramatically.

virtualphilharmony_a_conducting_system_with_heuristics_of_conducting_an_orchestra.txt · Last modified: 2015/12/17 21:59 (external edit)