=============================================================
Leonardo Electronic Almanac Volume 2, No. 9
September 1994
Craig Harris, Executive Editor
Roger Malina, Editor
Editorial Advisory Board
Roy Ascott, Michael Naimark, Simon Penny,
Greg Garvey, Joan Truckenbrod
ISSN #1071-4391
=============================================================
___________________
| |
| FEATURE ARTICLE |
|___________________|
( Visualizing Music & Sound - An Annotated Bibliography )
Craig Harris
718-6th Street SE
Minneapolis, MN 55414-1318
Tel: (612) 362-9390
Fax: (612) 362-0097
Email: harri067@maroon.tc.umn.edu
This article provides sources that represent work in the
area of visualizing music and sound. While not exhaustive,
this annotated bibliography offers substantial background
information with respect to the work that has taken place,
and is representative of some approaches that people have
taken in relating visual material to sonic/musical data.
The focus of this survey has been limited to material
relating to the visual representation of music and sound,
ignoring research and applications in such areas as
visualization of scientific data, even when the data is
sonic in nature. In addition, this work does not reflect
activity in the area of fusing sound and vision as it
relates to the human-computer interface, nor is it
concerned with music representation as it relates to
notation and sound editing resources.
This annotated bibliography is substantially enhanced by
perusal of the bibliographies which appear in the
individual articles (not duplicated here). In combination,
they provide insight into how people have thought about
merging sight and sound, and how these perspectives have
been implemented. Main sources consulted include the
journals "Leonardo", "Leonardo Music Journal", "Computer
Music Journal", and also the "Proceedings of the
International Computer Music Conferences", and the book
_The Visual Mind: Art and Mathematics_. A few commercial
and independent programs are mentioned.
I invite readers to send me comments and pointers to
additional sources at craig@well.sf.ca.us.
Leonardo Journal
----------------
Brill, Louis M., "Wavefronts Theatre of Performing Lights:
On Light Shows with Music and Dance", Leonardo 13, p. 307
(1980)
This article describes work which is based on creating
visual displays of light and patterns responding to the
intensity of sound produced by audio tape. This is not
a computer-based application, and is essentially a
performance environment resource.
Cook, Dick, "Kinetic Art: The Luminetic System With and
Without Sound", Leonardo 8, No. 1 (1975).
DeWitt, Tom, "Visual Music: Searching for an Aesthetic",
Leonardo 20, 115 (1987).
This is part of the literature which attempts to
address some of the aesthetic issues underlying the
expression of music in the visual domain. The article
is more theoretical in nature. It appears in the
bibliography as a reference to some of the issues being
considered on this topic.
Evans, Brian, "Establishing a Tonic Space with Digital
Color", Leonardo, Electronic Art Supplemental Issue, pp.
27-30 (1988).
Brian Evans relates the musical concept of tonal
centers to the visual realm in this article. This is a
theoretical article.
Galeyev, Bulat, "Farewell Prometheus Readings: Light Music
in the Former Soviet Union", Leonardo 27, No. 4, p. 351 (1994).
Galeyev describes 25 years of seminars, conferences and
festivals centering around "light music", held in
Kazan, and drawing participants from the entire Soviet
Union.
Grayson, John, "New Materials and Methods for the Musical
Instrument Designer, the Audio-Kinetic Sculptor, Musician
and Composer", Leonardo 3, p. 295 (1970).
Harrison, John and Simon Baron-Cohen, "Synaesthesia: An
Account of Coloured Hearing", Leonardo 27, No. 4, p. 343 (1994).
This article explores the condition of synaesthesia,
and provides some historical accounts. The authors
examine the impact upon the fields of art, music and
literature.
Jablonsky, Stephen, "Graphic Artworks Based on Music:
Musigraphs", Leonardo 12, p. 308 (1979).
Stephen Jablonsky creates graphic art works based on
music compositions. He attempts to address some of the
formal elements of music, and strives to characterize a
composition using the work's formal as well as poetic
characteristics. There is an attempt at value-added
information, inasmuch as he strives to create a visual
representation that somehow clarifies the work in a
unique way. His Musigraphs are two-dimensional, static
works, and there is no computer or specific new
technology used or implied.
Jack Ox with Peter Frank, "The Systematic Translation of
Musical Compositions into Paintings", Leonardo 17, No. 3,
p. 152 (1984).
As is the case with Stephen Jablonsky, Ox and Frank
characterize music compositions in a two-dimensional
visual framework - the painting. There is an attempt to
establish synesthetic linkage, to analyze music using
traditional theoretical techniques, to create a
systematic approach to visualizing the music, and to
incorporate intuitive means in selecting the visual
material. There is no computer mediation in the
process. This work embodies a personal approach to the
creation of static visual art, and is useful in terms
of understanding the individual nature of translating
music into the visual realm.
Justesen, Aage, "Pictures Based on Voice Graphs:
Pictonoms", Leonardo 11, p. 205 (1978)
Justesen's work is based on the transduction of vocal
sounds, and the creation of two-dimensional graphic art
works based on the images created in the process. In
addition to falling into the same category as Jablonsky
and Ox/Frank - characterizing sonic material in two-
dimensional, static visual art - Justesen creates a
direct sonic link by using the sound itself as the
catalyst or trigger for the visual representation.
Kostyniuk, Ron, "Electric Light Audio-Kinetic Artworks:
'Ecological Biomes'", Leonardo 12, p. 45 (1979).
Kostyniuk discusses his work fusing both visual and
aural elements in four-dimensional artwork - adding the
third dimension of his kinetic sculptures, with time -
the requisite for incorporating sound - acting as the
fourth dimension. This work is in the same vein as Cook
and Grayson. Kostyniuk further tries to establish an
organic fusion of elements, hence the characterization
as Ecological Biomes.
Nicolas Schoffer, "Sonic and Visual Structures: Theory and
Experiment", Leonardo 18, No. 2, p. 59 (1985).
There is both a theoretical and an application-oriented
aspect in Schoffer's work. He was primarily a visual
artist, approaching music from that vantage point in
creating sonic and multimedia environments. He did use
the computer in modeling his collective choreography,
and some of his work utilized recorded sound to affect
sculptural activity in the visual domain. This finds
its strength in the way that Schoffer envisions the
direct connection of sound and vision to create multi-
dimensional art works.
Peacock, Kenneth, "Instruments to Perform Color-Music: Two
Centuries of Technological Experimentation", Leonardo 21,
No. 4, pp. 397-406 (1988).
Peacock provides a detailed survey of technological
developments and specific implementations in the area
of associating colors to music source material. A
variety of projects are profiled, and Peacock describes
the various approaches that people have taken in
linking colors to sonic elements, to notation, and to
music instrument development. The article provides
valuable historical information, and I recommend
perusal, if only to be familiar with work in related
areas.
Pierre Y. Karinthi, "A Contribution to Musicalism: An
Attempt to Interpret Music in Painting", Leonardo 24, No.
4, p. 401 (1991).
Karinthi's work falls into the same category as
Jablonsky, Ox/Frank and Justesen, deriving two-
dimensional, static visual art from musical sources.
Karinthi establishes a mapping of musical elements to
the visual arts, including such components as pitch,
chord and timbre being mapped to color, graph and
intensity or luminosity, respectively. This is another
personal, systematic approach to visualizing classical
and romantic music. There is no new technology
suggested in his approach.
Pocock-Williams, Lynn, "Toward the Automatic Generation of
Visual Music", Leonardo 25, No.1, pp. 29-36 (1992)
In this article Lynn Pocock-Williams describes her
research involving the integration of sound and image.
She has created a computer software system to
automatically translate sound into image, using a rule-
based technology. This resulted in the development an
"abstract visual 'language', based upon color, linear
movement and geometric shapes, to give graphic
expression to the music" (p. 31 of her article). She
attempts to provide additional insights into the music
by representing abstract imagery which relates directly
to the music source. At the time of the writing of the
article Pocock-Williams was using only pitch and
duration in the music analysis, extracting MIDI data
and transforming that information through her own
system. Some of the components of this system include
the processing of MIDI data to create animated visual
representations of musical information; the automatic
nature of the process; and the attempt to characterize
some aspects of musical structure by linking musical
phrases with the phrases of animation. The visual
material appears to be selected from a library of
animations, suggesting that, at least in the
manifestation at the time of this article, the visual
animations have been previously generated, and are
arranged and/or transformed by the sound. The
orientation of the project seems to be towards an
aesthetically pleasing work of visual art, or
sonic/visual art, with a view towards synchronization
of the sonic and visual elements - not direct
generation of visual material from the sonic component.
Snyder, Robert R., "Video Color Control by Means of an
Equal-Tempered Keyboard", Leonardo 18, p. 93 (1985).
As the title suggests, color is controlled using a
keyboard controller, and the connection between the
visual and sonic elements is established through the
controller. In fact, the sonic element is nearly
incidental, and the key element is the use of a
controller which is ordinarily used for music as the
controlling element for visual effects.
Theo Goldberg and Gunther Schrack, "Computer-Aided
Correlation of Musical and Visual Structures", Leonardo
19, No. 1, p.11 (1986).
This article describes a project which has both a
theoretical dimension and a practical application. One
of the ways that Goldberg and Schrack achieve a
connection between the aural and the visual is by
developing a frequency-time distribution of sound
objects, representing them graphically by massaging the
data using a series of modeling, rendering and viewing
software modules. They describe how they attempt to
create an isomorphic relationship between aural and
visual structures. It seems that much of their work
creates visual art, some of which is static. The exact
nature of the music and its role is ambiguous, and is
seemingly described elsewhere in an article by Barry
Truax at Simon Fraser University. The work, as
represented in this article, does not reflect a deep
understanding of music.
Vadnerkar, S. V., "A Code for Representing the Occidental
Musical Notation in Pictorial Art", Leonardo 9, p. 140 (1976).
A system is proposed for relating music notation to
representation in the visual realm.
W. Garner, "The Relationship Between Colour and Music",
Leonardo 11, No. 3, p. 225 (1978).
This is a theoretical work which attempts to draw a
correlation between the octave and its division into
twelve semitones, and the spectrum of visible light. It
is listed here as another attempt to derive a
correlation between the senses using mathematical
principles.
Wagler, S. R., "'Sonovision': A Visual Display of Sound",
Leonardo 3, p. 443 (1970).
Wagler works with optical and electronic technology to
produce visual displays which are "color correlated to
sound by projecting a krypton or helium-neon laser
light beam on to a translucid screen or opaque
surface." The laser beam points towards a speaker, and
the vibrations produce the motion which the light beam
reflects on the screen. The method of transduction is
based on creating a visual reflection of a composite
audio signal. In this kind of representation there is
no knowledge or understanding of music voices,
independent pitches, or musical structure.
Whitney, Sr., John., "Fifty Years of Composing Computer
Music and Graphics: How Time's New Solid-State
Tractability Has Changed Audio-Visual Perspectives",
Leonardo 24, No. 5, p. 597 (1991).
This pioneer in computer graphics has written
extensively about color in motion, and correlations
with music/sound. As expressed in this article, his
work stresses algorithmic or generative processes. By
using the same processes to create both sonic and
visual material, Whitney maintains that a deep
connection is created. The experiments that he
conducted with his brother seem to be more oriented
towards proving and demonstrating principles, and not
in developing specific reproducible hardware
and/software.
Zajec, Edward, "Orphics: Computer Graphics and the Shaping
of Time with Color", Leonardo, Electronic Art Supplemental
Issue, pp. 111-116 (1988).
Zajec "discusses ideas and techniques for the fluid
articulation of color in time", and draws parallels
with the language of music. He raises such issues as
thematic transformation and modulation, and proposes a
language of light and sound. The work is primarily
theoretical, with some experimentation in one art work
described in the article. A correlation between aural
and visual material is being drawn, but the description
of a musical language is not sophisticated, and the
visual analog is abstract. There is a real time
component to the activity, but the forming principle
focuses on thematic transformation.
Leonardo Music Journal
----------------------
Evans, Brian, Elemental Counterpoint with Digital Imagery.
Leonardo Music Journal 2, pp. 13-18 (1992)
In this article Evans continues his work bridging the
aural and visual realms (see article above). Here he
proposes and illustrates a formal approach to abstract
animation based on contrapuntal music techniques. He
also compares his approach with some of the other work
represented in this bibliography (Whitney & Zajec), and
in that sense is a useful overview. The imagery is
abstract, as was the case with both Whitney & Zajec. To
the extent that one is attempting to create visual
representations of discrete musical lines, and to
establish a structural similarity in both aural and
visual realms, Evans' proposals are worth attention.
McLean, Barton, "Composition with Sound and Light",
Leonardo Music Journal 2, pp. 19-22 (1992)
This article describes the McLean Mix performance
system, which includes a MIDI-keyboard-controlled
Sparkling Light Console (SLC). This system produces
"abundantly varied colored patterns of pulsed lights in
linear and textural configurations", linked with a MIDI
music system. This allows the musician to create MIDI
control signals, using that information to create maps
onto the control of the light system. There also seems
to be the possibility for creating independent mappings
on the keyboard, with regard to any of the purely
musical aspects of the keyboard design. McLean
maintains that this work invites "direct analogies
between a single discrete point of light and a single
discrete musical tone, between the rhythm of a light
pattern and the rhythm of a musical phrase, and between
the color of light and the timbre of a musical tone".
The imagery is abstract patterns of projected light.
There is no attempt to correlate pitch to a position on
a screen, for example. Furthermore, since this is a
performance system, the interests in drawing a
correlation between sight and sound concentrate on
creating an aesthetically pleasing effect. There is no
attempt to portray the visual material as a musical
representation of the composition.
The Visual Mind: Art and Mathematics
Goffredo Haus and Paolo Morini, "TEMPER: A System for
Music Synthesis from Animated Tessellations", in The
Visual Mind: Art and Mathematics, A Leonardo Book, pp.
171-176 (Cambridge, MA: The MIT Press).
This work utilizes mathematical principles to make the
correlation between aural and visual realms. This is
similar to Whitney's approach, for example, in that
there is a fundamental algorithm or generative
component which attempts to link sight and sound
directly.
International Computer Music Conference (ICMC) Proceedings
Demers, Louis-Philippe, "A Performance Instrument for
Lighting", Proceedings of the 1991 International Computer
Music Conference, pp. 471-474 (San Francisco, CA:
International Computer Music Association).
Demers' work is important because it is an automated
system based on the control of a lighting performance
instrument using MIDI. This concept is similar to the
McLean work described above, but Demers' system
provides an interactive graphical programming
environment. This approach facilitates the audio and
visual linkage, but the orientation is still primarily
towards the algorithmic or generative musical
principles. The correlation with the musical material,
or the material from which both the aural and visual
material are derived, can be extremely direct, and as
explicit as required by the circumstance. However, it
is still a stage lighting performance system, designed
to create a visual effect on the stage which relates to
the musical material performed. This is not notation,
nor can it really be viewed as music representation in
its current application.
Mont-Reynaud, Bernard, "SeeMusic: A Tool for Music
Visualization", Proceedings of the 1993 International
Computer Music Conference, pp. 457-460 (San Francisco, CA:
International Computer Music Association).
Bernard Mont-Reynaud describes his work that began at
Stanford University's CCRMA, and continued at Studer
Editech, a company which builds audio recording and
studio editing resources. In creating SeeMusic, the
"goal was to build a multi-purpose visualization tool
to serve the needs of auditory perception research and
its relation to the development of new applications in
music and audio". Mont-Reynaud takes an approach which
analyzes the musical signal, and builds a paradigm for
interpreting that signal, transforming it into visual
representations using a synthesis by convolution
procedure. This method is combined with what the author
calls the dual representation - a spectral image and a
pitch image - "with user-controlled convolution
relations between the two. Generally speaking, this
provides a correspondence map between sound data,
viewed as log-frequency spectrograms, and MIDI data or
musical score data, viewed on the same coordinate
system."
Computer Music Journal
----------------------
Whitney, Sr., John., "To Paint on Water: The Audiovisual
Duet of Complementarity", Computer Music Journal 18, No.
3, p. 45 (1994).
Whitney provides an update on his work merging music
with visual art. The complementarity refered to in this
article is based on a common foundation of harmony,
resulting in a new art form, where "one can design and
execute visual and musical patterns in an inter-
reactive form of temporal union". Whitney uses his own
specialized program, developed by his associate Jerry
Reed, which he suggests is functional as a real-time
performance instrument controlling both sound and
graphics.
Programs and Products
---------------------
Lightworks Graphics Synthesizer
138 North Main Street, Suite 111
Sebastopol, CA 95472
Tel: (800) 892-3325
This is an Amiga-based graphics synthesizer which
builds a studio editing and performance system using
MIDI. The Lightworks Graphics Synthesizer does not
correlate music score information with a music
representation. The orientation is more towards
establishing a flexible environment to create visual or
audio-visual art works. The literature suggests that
there is much control over the visual representations,
and in some way it may be possible to create color and
geometric displays which relate to musical structure.
Here as well, by using MIDI as the control protocol,
there is a relatively facile and standardized method
for establishing the audio and visual analogs. There is
further suggestion that one's music could be better
comprehended by using the synthesizer to graphically
interpret the music. There is no explicit mention in
the literature to anything that would indicate that
there is any musical understanding inherent in the
system.
Music Animation Machine
Stephen Malinowski
P.O. Box 13622
Berkeley, CA 94701
Stephen Malinowski's Music Animation Machine (MAM) is
essentially a system for preparing piano roll style,
animated visual representations of musical scores. This
is another MIDI-based system using a
note/duration/velocity/track(voice) paradigm to create
pitch-based, colorized representations of music. In
some of Malinowski's examples color relates to a
specific track or voice within the composition. In
other instances he relates color to relative amplitude,
in an attempt to reflect different characteristics of
the music. There is nothing in the Music Animation
Machine which reflects changing the color intensity
within a musical note - notes are either on or off.
While there is mention in the MAM of an attempt to
visually represent musical structure, there doesn't
seem to be anything reflected in the music examples
which does anything other than the most trivial,
graphically-observable structure in the
representations.
Back to Dr. Paradise's Main Page