Ten Years of Automatic Mixing


Automatic microphone mixers have been around since 1975. These are devices that lower the levels of microphones that are not in use, thus reducing background noise and preventing acoustic feedback. They’re great for things like conference settings, where there may be many microphones but only a few speakers should be heard at any time.

Over the next three decades, various designs appeared, but it didn’t really grow much from Dan Dugan’s original Dan Dugan’s original concept.

Enter Enrique Perez Gonzalez, a PhD student researcher and experienced sound engineer. On September 11th, 2007, exactly ten years ago from the publication of this blog post, he presented a paper “Automatic Mixing: Live Downmixing Stereo Panner.” With this work, he showed that it may be possible to automate not just fader levels in speech applications, but other tasks and for other applications. Over the course of his PhD research, he proposed methods for autonomous operation of many aspects of the music mixing process; stereo positioning, equalisation, time alignment, polarity correction, feedback prevention, selective masking minimization, etc. He also laid out a framework for further automatic mixing systems.

Enrique established a new field of research, and its been growing ever since. People have used machine learning techniques for automatic mixing, applied auditory neuroscience to the problem, and explored where the boundaries lie between the creative and technical aspects of mixing. Commercial products have arisen based on the concept. And yet all this is still only scratching the surface.

I had the privilege to supervise Enrique and have many anecdotes from that time. I remember Enrique and I going to a talk that Dan Dugan gave at an AES convention panel session and one of us asked Dan about automating other aspects of the mix besides mic levels. He had a puzzled look and basically said that he’d never considered it. It was also interesting to see the hostile reactions from some (but certainly not all) practitioners, which brings up lots of interesting questions about disruptive innovations and the threat of automation.


Next week, Salford University will host the 3rd Workshop on Intelligent Music Production, which also builds on this early research. There, Brecht De Man will present the paper ‘Ten Years of Automatic Mixing’, describing the evolution of the field, the approaches taken, the gaps in our knowledge and what appears to be the most exciting new research directions. Enrique, who is now CTO of Solid State Logic, will also be a panellist at the Workshop.

Here’s a video of one of the early Automatic Mixing demonstrators.

And here’s a list of all the early Automatic Mixing papers.

  • E. Perez Gonzalez and J. D. Reiss, A real-time semi-autonomous audio panning system for music mixing, EURASIP Journal on Advances in Signal Processing, v2010, Article ID 436895, p. 1-10, 2010.
  • Perez-Gonzalez, E. and Reiss, J. D. (2011) Automatic Mixing, in DAFX: Digital Audio Effects, Second Edition (ed U. Zölzer), John Wiley & Sons, Ltd, Chichester, UK. doi: 10.1002/9781119991298. ch13, p. 523-550.
  • E. Perez Gonzalez and J. D. Reiss, “Automatic equalization of multi-channel audio using cross-adaptive methods”, Proceedings of the 127th AES Convention, New York, October 2009
  • E. Perez Gonzalez, J. D. Reiss “Automatic Gain and Fader Control For Live Mixing”, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, New York, October 18-21, 2009
  • E. Perez Gonzalez, J. D. Reiss “Determination and correction of individual channel time offsets for signals involved in an audio mixture”, 125th AES Convention, San Francisco, USA, October 2008
  • E. Perez Gonzalez, J. D. Reiss “An automatic maximum gain normalization technique with applications to audio mixing.”, 124th AES Convention, Amsterdam, Netherlands, May 2008
  • E. Perez Gonzalez, J. D. Reiss, “Improved control for selective minimization of masking using interchannel dependency effects”, 11th International Conference on Digital Audio Effects (DAFx), September 2008
  • E. Perez Gonzalez, J. D. Reiss, “Automatic Mixing: Live Downmixing Stereo Panner”, 10th International Conference on Digital Audio Effects (DAFx-07), Bordeaux, France, September 10-15, 2007

Female pioneers in audio engineering

The Heyser lecture is a distinguished talk given at each AES Convention by eminent individuals in audio engineering and related fields. At the 140th AES Convention, Rozenn Nicol was the Heyser lecturer. This was well-deserved, and she has made major contributions to the field of immersive audio. But what was shocking about this is that she is the first woman Heyser lecturer. Its an indicator that woman are under-represented and under-recognised in the field. With that in mind, I’d like to highlight some women who have made major contributions to the field, especially in research and innovation.

  • Birgitta Berglund led major research into the impact of noise on communities. Her influential research resulted in guidelines from the World Health Organisation, and greatly advanced our understanding of noise and its effects on society. She was the 2009 IOA Rayleigh medal recipient.
  • Marina Bosi is a past AES president of the AES. She has been instrumental in the development of standards for audio coding and digital content management standards and formats, including develop the AC-2, AC-3, and MPEG-2 Advanced Audio Coding technologies,
  • Anne-Marie Bruneau has been one of the most important researchers on electrodynamic loudspeaker design, exploring motion impedance and radiation patterns, as well as establishing some of the main analysis and measurement approaches used today. She co-founded the Laboratoire d’Acoustique de l’Université du Maine, now a leading acoustics research center.
  • Ilene J. Busch-Vishniac is responsible for major advances in the theory and understanding of electret microphones, as well as patenting several new designs. She received the ASA R. Bruce Lindsay Award in 1987, and the Silver Medal in Engineering Acoustics in 2001. President of the ASA 2003-4.
  • Elizabeth (Betsy) Cohen was the first female president of the Audio Engineering Society. She was presented with the AES Fellowship Award in 1995 for contributions to understanding the acoustics and psychoacoustics of sound in rooms. In 2001, she was presented with the AES Citation Award for pioneering the technology enabling collaborative multichannel performance over the broadband internet.
  • crumPoppy Crum is head scientist at Dolby Laboratories whose research involves computer research in music and acoustics. At Dolby, she is responsible for integrating neuroscience and knowledge of sensory perception into algorithm design, technological development, and technology strategy.
  • Delia Derbyshire (1937-2001) was an innovator in electronic music who pushed the boundaries of technology and composition. She is most well-known for her electronic arrangement of the theme for Doctor Who, an important example of Musique Concrète. Each note was individually crafted by cutting, splicing, and stretching or compressing segments of analogue tape which contained recordings of a plucked string, oscillators and white noise. Here’s a video detailing a lot of the effects she used, which have now become popular tools in digital music production.
  •  Ann Dowling is the first female president of the Royal Academy of Engineering. Her research focuses on noise analysis and reduction, especially from engines, and she is a leading educator in acoustics. A quick glance at google scholar shows how influential her research has been.
  • Marion Downs was an audiometrist at Colorado Medical Center in Denver, who invented the tests used to measure hearing both In newly born babies and in fetuses.
  • Judy Dubno is Director of Hearing Research at the Medical University of South Carolina. Her research focuses on human auditory function, with emphasis on the processing of auditory information and the recognition of speech, and how these abilities change in adverse listening conditions, with age, and with hearing loss. Recipient of the James Jerger Career Award for Research in Audiology from the American Academy of Audiology and Carhart Memorial Lecturer for the American Auditory Society. President of the ASA in 2014-15.
  • thumb_FiebrinkPhoto3Rebecca Fiebrink researches Human Computer Interaction (HCI) and its application of machine learning to real-time, interactive, and creative domains. She is the creator of the popular Wekinator, which allows anyone to use machine learning to build new musical instruments, real-time music information retrieval and audio analysis systems, computer listening systems and more.
  • Katherine Safford Harris pioneered EMG studies of speech production and auditory perception. Her research was fundamental to speech recognition, speech synthesis, reading machines for the blind, and the motor theory of speech perception. She was elected Fellow of the ASA, the AAAS, the American Speech-Language-Hearing Association, and the New York Academy of Sciences. She was President of the ASA (2000-2001), awarded the Silver Medal in 2005 and Gold Medal in 2007.
  • Rhona Hellman was a Fellow of the ASA. She was a distinguished hearing scientist and preeminent expert in auditory perceptual phenomena. Her research spanned almost 50 years, beginning in 1960. She tackled almost every aspect of loudness, and the work resulted in major advances and developments of loudness standards.
  • Mara Helmuth developed software for composition and improvisation involving granular synthesis. Throughout the 1990s, she paved the way forward by exploring and implementing systems for collaborative performance over the Internet. From 2008-10 she was President of the International Computer Music Association.
  • Carleen_HutchinsCarlene Hutchins (1911-2009) was a leading researcher in the study of violin acoustics, with over a hundred publications in the field. She was founder and president of the Catgut Society, an organization devoted to the study and appreciation of stringed instruments .
  • Sophie Germain (1776-1831) was a French mathematician, scientist and philosopher. She won a major prize from the French Academy of Sciences for developing a theory to explain the vibration of plates due to sound. The history behind her contribution, and the reactions of leading French mathematicians to having a female of similar calibre in their midst, is fascinating. Joseph Fourier, whose work underpins much of audio signal processing, was a champion of her work.
  • Bronwyn Jones was a psychoacoustician at the CBS Technology Center during the 70s and 80s. In seminal work with co-author Emil Torrick, she developed one of the first loudness meters, incorporating both psychoacoustic principles and detailed listening tests. It paved the way for what became major initiatives for loudness measurement, and in some ways outperforms the modern ITU 1770 standard
  • Bozena Kostek is editor of the Journal of the Audio Engineering Society. Her most significant contributions include the applications of neural networks, fuzzy logic and rough sets to musical acoustics, and the application of data processing and information retrieval to the psychophysiology of hearing. Her research has garnered dozens of prizes and awards.
  • Daphne Oram (1925 –2003) was a pioneer of ‘musique concrete’ and a central figure in the evolution of electronic music. She devised the Oramics technique for creating electronic sounds, co-founded the BBC Radiophonic Workshop, and was possibly the first woman to direct an electronic music studio, to set up a personal electronic music studio and to design and construct an electronic musical instrument.
  • scalettiCarla Scaletti is an innovator in computer generated music. She designed the Kyma sound generation computer language in 1986 and co-founded Symbolic Sound Corporation in 1989. Kyma is one of the first graphical programming languages for real time digital audio signal processing, a precursor to MaxMSP and PureData, and is still popular today.
  • Bridget Shield was professor of acoustics at London Southbank University. Her research is most significant in our understanding of the effects of noise on children, and has influenced many government initiatives. From 2012-14, she was the first female President of the Institute of Acoustics.
  • Laurie Spiegel created one of the first computer-based music composition programs, Music Mouse: an Intelligent Instrument, which also has some early examples of algorithmic composition and intelligent automation, both of which are hot research topics today.
  • maryMary Desiree Waller (1886-1959) wrote a definitive treatise on Chladni figures, which are the shapes and patterns made by surface vibrations due to sound, see Sophie Germain, above. It gave far deeper insight into the figures than any previous work.
  • Megan (or Margaret) Watts-Hughes is the inventor of the Eidophone, an early instrument for visualising the sounds made by your voice. She rediscovered this simple method of generating Chladni figures without knowledge of Sophie Germain or Ernst Chladni’s work. There is a great description of her experiments and analysis in her own words.

The Eidophone, demonstrated by Grace Digney.

Do you know some others who should be mentioned? We’d love to hear your thoughts.

Thanks to Theresa Leonard for information on past AES presidents. She was the third female president.  will be the fourth.

And check out Women in Audio: contributions and challenges in music
technology and production for a detailed analysis of the current state of the field.

Sound as a Weapon

Sonic weapons frequently occur in science fiction and fantasy. I remember reading the Tintin book The Calculus affair, where Professor Calculus invents ultrasonic devices which break glass objects around the house. But the bad guys from Borduria want to make them large scale and long range devices, capable of mass destruction.

As with many fantastic fiction ideas, sonic weapons have a firm basis in fact. But one of the first planned uses for sonic devices in war was as a defense system, not a weapon.

Between about 1916 and 1936, acoustic mirrors were built and tested around the coast of England. The idea is that they could reflect, and in some cases focus, the sound of incoming enemy aircraft. Microphones could be placed at the foci of the reflectors, giving listeners a means of early detection. The mirrors were usually parabolic or spherical in shape detect the aircraft, and for the spherical designs, the microphone could be moved as a means of identifying the direction of arrival.


It was a good idea at first, but air speed of bombers and fighters improved so much over that time period that it would only give a few minutes extra warning. And then the technology became completely obsolete with the invention of radar, though that also meant that the effort into planning a network of detectors along the coast was not wasted.

The British weren’t the only ones attempting to use sound for aircraft detection between the world wars. The Japanese had mobile acoustic locators known as ‘war tubas,’ Dutch had personal horns and personal parabolas, the Czechs used a four-horn acoustic locator to detect height as well as horizontal direction, and the French physicist Jean-Baptiste Perrin designed the télésitemètre, which in a field full of unusual designs, still managed to distinguish itself by having 36 small hexagonal horns. Perrin though, is better known for his Nobel prize winning work on Brownian motion that finally confirmed the atomic theory of matter. Other well-known contributors to the field include the Austrian born ethnomusicologist Erich Moritz von Hornbo and renowned psychologist Max Wertheimer. Together, they developed the sound directional locator known as the Wertbostel, which was believed to have been commercialised during the 30s.
There are wonderful photos of these devices, most of which can be found here , but I couldn’t resist including at least a couple,

german%201917a a German  acoustic & optical locating apparatus, and a Japanese war tuba.

hiro1a and a Japanese war tuba.

But these acoustic mirrors and related systems were all intended for defense. During World War II, German scientists worked on sonic weapons under the supervision of Albert Speer. They developed an acoustic cannon that was intended to send a deafening, focused beam of sound, magnified by parabolic reflector dishes. Research was discontinued however, since initial efforts were not successful, nor was it likely to be effective in practical situations.

Devices capable of producing especially loud sounds, often focused in a given direction or over a particular frequency range, have found quite a few uses as weapons of some kind. A long-range acoustic device was used to deter pirates who attempted to  attack a cruise ship, for instance, and sonic devices emitting high frequencies that might be heard by teenagers but unlikely to be heard by adults have been deployed in city centres to prevent youth from congregating. However, such stories make for interesting reading, but it’s hard to say how effective they actually are.
And there are even sonic weapons occurring in nature.

The snapping shrimp has a claw which shoots a jet of water, which in turn generates a cavitation bubble. The bubble bursts with a snap reaching around 190 decibels. Its loud enough to kill or stun small sea creatures, who then become its prey.

John Cage and the anechoic chamber


An acoustic anechoic chamber is a room designed to be free of reverberation (hence non-echoing or echo-free). The walls, ceiling and floor are usually lined with a sound absorbent material to minimise reflections and insulate the room from exterior sources of noise. All sound energy will travel away from the source with almost none reflected back. Thus a listener within an anechoic chamber will only hear the direct sound, with no reverberation.

The anechoic chamber effectively simulates a quiet open-space of infinite dimension. Thus, they are used to conduct acoustics experiments in ‘free field’ conditions. They are often used to measure the radiation pattern of a microphone or of a noise source, or the transfer function of a loudspeaker.

An anechoic chamber is very quiet, with noise levels typically close to the threshold of hearing in the 10–20 dBA range (the quietest anechoic chamber has a decibel  level of -9.4dBA, well below hearing). Without the usual sound cues, people find the experience of being in an anechoic chamber very disorienting and often lose their balance. They also sometimes detect sounds they would not normally perceive, such as the beating of their own heart.

One of the earliest anechoic chambers was designed and built by Leo Beranek and Harvey Sleeper in 1943. Their design is the one upon which most modern anechoic chambers is based. In a lecture titled ‘Indeterminacy,’ the avant-garde composer John Cage described his experience when he visited Beranek’s chamber.

“in that silent room, I heard two sounds, one high and one low. Afterward I asked the engineer in charge why, if the room was so silent, I had heard two sounds… He said, ‘The high one was your nervous system in operation. The low one was your blood in circulation.’”

After that visit, he composed his famous work entitled 4’33”, consisting solely of silence and intended to encourage the audience to focus on the ambient sounds in the listening environment.

In his 1961 book ‘Silence,’ Cage expanded on the implications of his experience in the anechoic chamber. “Try as we might to make silence, we cannot… Until I die there will be sounds. And they will continue following my death. One need not fear about the future of music.”

The beginning of stereo

5a9cc9_6da9661bf6bc4c6bbc8d49e310139509 Alan and Doreen Blumlein wedding photo

The sound reproduction systems for the early ‘talkie’ movies  often had only a single loudspeaker. Because of this, the actors all sounded like they were in the same place, regardless of their position on screen.

In 1931, the electronics and sound engineer Alan Blumlein and his wife Doreen went to see a movie where this monaural sound reproduction occured. According to Doreen, as they were leaving the cinema, Alan said to her ‘Do you realise the sound only comes from one person?’  And she replied, ‘Oh does it?’  ‘Yes.’ he said, ‘And I’ve got a way to make it follow the person’.

The genesis of these ideas is uncertain (though it might have been while watching the movie), but he described them to Isaac Shoenberg, managing director at EMI and Alan’s mentor, in the late summer of 1931. Blumlein detailed his stereo technology in the British patent “Improvements in and relating to Sound-transmission, Sound-recording and Sound-reproducing systems,” which was accepted June 14, 1933.


The serendipitous invention of the wah-wah pedal


In a previous post, we discussed some creative uses of the wah-wah creative uses of the wah-wah effect.

The first wah-wah pedal is attributed to Brad Plunkett in 1966, who worked at Warwick Electronics Inc., which owned Thomas Organ Company. Warwick Electronics acquired the Vox name due to the brand name’s popularity and association with the Beatles. Their subsidiary, Thomas Organ Company, needed a modified design for the Vox amplifier, which had a midrange boost, so that it would be less expensive to manufacture.

In a 2005 interview (M. Vdovin, “Artist Interview: Brad Plunkett,” Universal Audio WebZine, vol. 3, October 2005) Brad Plunkett said, I “came up with a circuit that would allow me to move this midrange boost … As it turned out, it sounded absolutely marvelous while you were moving it. It was okay when it was standing still, but the real effect was when you were moving it and getting a continuous change in harmonic content. We turned that on in the lab and played the guitar through it… I turned the potentiometer and he played a couple licks on the guitar, and we went crazy.

A couple of years later… somebody said to me one time, ‘You know Brad, I think that thing you invented changed music.’”

Acoustic reverberators

Today, reverb is most often added to a recording using artificial reverberators, such as software plug-ins or digital reverb hardware units. But there are a lot of other approaches.

Many recording studios have used special rooms known as reverberation chambers to add reverb to a performance. Elevator shafts and stairwells (as in New York City’s Avatar Recording Studio) work well as highly reverberant rooms. The reverb can also be controlled by adding absorptive materials like curtains and rugs.

Spring reverbs are found in many guitar amplifiers and have been used in Hammond organs. The audio signal is coupled to one end of the spring by a transducer that creates waves traveling through the spring. At the far end of the spring, another transducer converts the motion of the string into an electrical signal, which is then added to the original sound. When a wave arrives at an end of the spring, part of the wave’s energy is reflected. However, these reflections have different delays and attenuations from what would be found in a natural acoustic environment, and there may be some interaction between the waves in a spring, thus this results in a slightly unusual (though not unpleasant) reverb sound.

Often several springs with different lengths and tensions are enclosed in a metal box, known as the reverb pan, and used together. This avoids uniform behavior and creates a more realistic, pseudorandom series of echoes. In most reverb units though, the spring lengths and tensions are fixed in the design process, and not left to the user to control.

The plate reverb is similar to a spring reverb, but instead of springs, the  transducers are attached at several locations on a metal plate. These transducers send vibrations through the plate, and reflections are produced whenever a wave reaches the plate’s edge. The location of the transducers and the damping of the plate can be adjusted to control the reverb. However, plate reverbs are expensive and bulky, and hence not widely used.

Water tank reverberators have also been used. Here, the audio signal is modulated with an ultrasonic signal and transmitted through a tank of water. The output is then demodulated, resulting in the reverberant output sound. Other reverberators include pipes with microphones placed at various points.

These acoustic and analogue reverberators can be interesting to create and use, but they lack the simplicity and ease of use of digital reverberators. Ultimately, the choice of implementation is a matter of taste.