Sound Talking at the Science Museum featured assorted speakers on sonic semantics

sound-talking-logo-large

On Friday 3 November, Dr Brecht De Man (Centre for Digital Music, Queen Mary University of London) and Dr Melissa Dickson (Diseases of Modern Life, University of Oxford) organised a one-day workshop at the London Science Museum on the topic of language describing sound, and sound emulating language. We discussed it in a previous blog entry, but now we can wrap up and discuss what happened.

Titled ‘Sound Talking‘, it brought together a diverse lineup of speakers around the common theme of sonic semantics. And with diverse we truly mean that: the programme featured a neuroscientist, a historian, an acoustician, and a Grammy-winning sound engineer, among others.

The event was born from a friendship between two academics who had for a while assumed their work could not be more different, with music technology and history of Victorian literature as their respective fields. When learning their topics were both about sound-related language, they set out to find more researchers from maximally different disciplines and make it a day of engaging talks.

After having Dr Dickson as a resident researcher earlier this year, the Science Museum generously hosted the event, providing a very appropriate and ‘neutral’ central London venue. The venue was further supported by the Diseases of Modern Life project, funded by the European Research Council, and the Centre for Digital Music at Queen Mary University of London.

The programme featured (in order of appearance):

  • Maria Chait, Professor of auditory cognitive neuroscience at UCL, on the auditory system as the brain’s early warning system
  • Jonathan Andrews, Reader in the history of psychiatry at Newcastle University, on the soundscape of the Bethlehem Hospital for Lunatics (‘Bedlam’)
  • Melissa Dickson, postdoctoral researcher in Victorian literature at University of Oxford, on the invention of the stethoscope and the development of an associated vocabulary
  • Mariana Lopez, Lecturer in sound production and post production at University of York, on making film accessible for visually impaired audiences through sound design
  • David M. Howard, Professor of Electronic Engineering at Royal Holloway University of London, on the sound of voice and the voice of sound
  • Brecht De Man, postdoctoral researcher in audio engineering at Queen Mary University of London, on defining the language of music production
  • Mandy Parnell, mastering engineer at Black Saloon Studios, on the various languages of artistic direction
  • Trevor Cox, Professor of acoustic engineering at University of Salford, on categorisation of everyday sounds

In addition to this stellar speaker lineup, Aleks Kolkowski (Recording Angels) exhibited an array of historic sound making objects, including tuning forks, listening tubes, a monochord, and a live recording of a wax cylinder. The workshop took place in a museum, after all, where Dr Kolkowski has held a research associateship, so the display was very fitting.

The full program can be found on the event’s web page. Video proceedings of the event are forthcoming.

Advertisements

Applied Science Journal Article

We are delighted to announce the publication of our article titled, Sound Synthesis of Objects Swinging through Air Using Physical Models in the Applied Science Special Issue on Sound and Music Computing.

 

The Journal is a revised and extended version of our paper which won a best paper award at the 14th Sound and Music Computing Conference which was held in Espoo, Finland in July 2017. The initial paper presented a physically derived synthesis model used to replicate the sound of sword swings using equations obtained from fluid dynamics, which we discussed in a previous blog entry. In the article we extend listening tests to include sound effects of metal swords, wooden swords, golf clubs, baseball bats and broom handles as well as adding in a cavity tone synthesis model to replicate grooves in the sword profiles. Further test were carried out to see if participants could identify which object our model was replicating by swinging a Wii Controller.
The properties exposed by the sound effects model could be automatically adjusted by a physics engine giving a wide corpus of sounds from one simple model, all based on fundamental fluid dynamics principles. An example of the sword sound linked to the Unity game engine is shown in this video.
 

 

Abstract:
A real-time physically-derived sound synthesis model is presented that replicates the sounds generated as an object swings through the air. Equations obtained from fluid dynamics are used to determine the sounds generated while exposing practical parameters for a user or game engine to vary. Listening tests reveal that for the majority of objects modelled, participants rated the sounds from our model as plausible as actual recordings. The sword sound effect performed worse than others, and it is speculated that one cause may be linked to the difference between expectations of a sound and the actual sound for a given object.
The Applied Science journal is open access and a copy of our article can be downloaded here.

Bounce, bounce, bounce . . .

bounce

Another in our continuing exploration of everyday sounds (Screams, Applause, Pouring water) is the bouncing ball. It’s a nice one for a blog entry since there are only a small number of papers focused on bouncing, which means we can give a good overview of the field. It’s also one of those sounds that we can identify very clearly; we all know it when we hear it. It has two components that can be treated separately; the sound of a single bounce and the timing between bounces.

Let’s consider the second aspect. If we drop a ball from a certain height and ignore any drag, the time it takes to hit the ground is completely determined by gravity. When it hits the ground, some energy is absorbed on impact. And so it may be traveling downwards with a velocity v1 just before impact, and after impact travels upwards with velocity v2. The ratio v2/v1 is called the coefficient of restitution (COR). A high COR means that the ball travels back up almost to its original height, and a low COR means that most energy is absorbed and it only travels up a short distance.

Knowing COR, one can use simple equations of motion to determine the time between each bounce. And since the sum of the times between bounces is a convergent series, one can find the maximum time until it stops bouncing. Conversely, measuring the coefficient of friction from times between bounces is literally a tabletop physics experiment (Aguiar 2003, Farkas 2006, Schwarz 2013). And kinetic energy depends on the square of the velocity, so we know how much energy is lost with each bounce, which also gives an idea of how the sound levels of successive bounces should decrease.

[The derivation of all this has been left to the reader 😊. But again, its straightforward application of the equations of motion that give time dependence of position and velocity under constant acceleration]

Its not that hard to extend this approach, for instance by including air drag or sloped surfaces. But if you put the ball on a vibrating platform, all sorts of wonderful nonlinear behaviour can be observed; chaos, locking and chattering (Luck 1993).

For instance, have a look at the following video; which shows some interesting behaviour where bouncing balls all seem to organise onto one side of a partition.

So much for the timing of bounces, but what about the sound of a single bounce? Well, Nagurka (2004) modelled the bounce as a mass-spring-damper system, giving the time of contact for each bounce. It provides a little more realism by capturing some aspects of the bounce sound, Stoelinga (2007) did a detailed analysis of bouncing and rolling sounds. It has a wealth of useful information, and deep insights into both the physics and perception of bouncing, but stops short of describing how to synthesize a bounce.

To really capture the sound of a bounce, something like modal synthesis should be used. That is, one should identify the modes that are excited for impact of a given ball on a given surface, and their decay rates. Farnell measured these modes for some materials, and used those values to synthesize bounces in Designing Sound . But perhaps the most detailed analysis and generation of such sounds, at least as far as I’m aware, is in the work of Davide Rocchesso and his colleagues, leaders in the field of sound synthesis and sound design. They have produced a wealth of useful work in the area, but an excellent starting point is The Sounding Object.

Are you aware of any other interesting research about the sound of bouncing? Let us know.

Next week, I’ll continue talking about bouncing sounds with discussion of ‘the audiovisual bounce-inducing effect.’

References

  • Aguiar CE, Laudares F. Listening to the coefficient of restitution and the gravitational acceleration of a bouncing ball. American Journal of Physics. 2003 May;71(5):499-501.
  • Farkas N, Ramsier RD. Measurement of coefficient of restitution made easy. Physics education. 2006 Jan;41(1):73.
  • Luck, J.M. and Mehta, A., 1993. Bouncing ball with a finite restitution: chattering, locking, and chaos. Physical Review E, 48(5), p.3988.
  • Nagurka, M., Shuguang H,. “A mass-spring-damper model of a bouncing ball.” American Control Conference, 2004. Vol. 1. IEEE, 2004.
  • Schwarz O, Vogt P, Kuhn J. Acoustic measurements of bouncing balls and the determination of gravitational acceleration. The Physics Teacher. 2013 May;51(5):312-3.
  • Stoelinga C, Chaigne A. Time-domain modeling and simulation of rolling objects. Acta Acustica united with Acustica. 2007 Mar 1;93(2):290-304.

Sound Talking – 3 November at the London Science Museum

On Friday 3 November 2017, Dr Brecht De Man (one of the audio engineering group researchers) and Dr Melissa Dickson are chairing an unusual and wildly interdisciplinary day of talks, tied together by the theme ‘language describing sound, and sound emulating language’.

Despite being part of the Electronic Engineering and Computer Science department, we think about and work around language quite a lot. After all, audio engineering is mostly related to transferring and manipulating (musical, informative, excessive, annoying) sound and therefore we need to understand how it is experienced and described. This is especially evident from projects such as the SAFE plugins, where we collect terms which describe a particular musical signal manipulation, to then determine their connection with the chosen process parameters and measured signal properties. So the relationship between sound and language is actually central to Brecht’s research, as well as of others here.

The aim of this event is to bring together a wide range of high-profile researchers who work on this intersection, from maximally different perspectives. They study the terminology used to discuss sound, the invention of words that capture sonic experience, and the use and manipulation of sound to emulate linguistic descriptions. Talks will address singing voice research, using sound in accessible film for hearing impaired viewers, new music production tools, auditory neuroscience, sounds in literature, the language of artistic direction, and the sounds of the insane asylum. ‘Sounds’ like a fascinating day at the Science Museum!

Register now (the modest fee just covers lunch, breaks, and wine reception) and get to see

  • Maria Chait (head of UCL Auditory Cognitive Neuroscience lab)
  • Jonathan Andrews (on soundscape of the insane asylum)
  • Melissa Dickson (historian of 19th century literature)
  • Mariana Lopez (making film more accessible through sound)
  • David Howard (the singing voice)
  • Brecht De Man (from our group, on understanding the vocabulary of mixing)
  • Mandy Parnell (award winning mastering engineer)
  • Trevor Cox (categorising quotidian sounds)

In addition, there will be a display of cool sound making objects, with a chance to make your own wax cylinder recording, and more!

The full programme including abstracts and biographies can be found on www.semanticaudio.co.uk/events/soundtalking/.

Aural fabric

This is a slightly modified version of a post that originally appeared on the Bela blog.

Alessia Milo is an architect currently researching education in acoustics for architecture while pursuing her PhD  with the audio engineering team here, as well as with the Media and Arts Technology programme.

She will present Influences of a Key Map on Soundwalk Exploration with a Textile Sonic Map at the upcoming AES Convention.

Here, she  introduces Aural Fabric, a captivating interactive sound installation consisting of a textile map which plays back field recordings when touched.

Aural Fabric is an interactive textile map allowing you to listen to selected field recordings by touching areas of the map that can sense touch. It uses conductive thread, capacitive sensing and Bela to process sensor data and play back the field recordings. The first map that was made represents a selection of sounds from the area of Greenwich, London. The field recordings of the area were captured with binaural microphones as part of a group soundwalk as part of a study on sonic perception. For the installation I chose recordings of particular locations that have a unique sonic identity, which you can listen to here. The textile map was created as a way of presenting these recordings to the general public.

When I created this project I wanted people to be able to explore the fabric surface of the map and hear the field recordings of the specific locations on the map as they touched it. An interesting way to do this was with conductive thread that I could embroider into the layout of the map. To read the touches from the conductive areas of the map I decided to use the MPR121 capacitive touch sensing board along with a Bela board.

Designing the map

 

I first considered the scale of the map based on how big the conductive areas could be in order to be touched comfortably, and on the limits of the embroidery machine used (Brother Pr1000E) . I finally settled on a 360mmx200mm frame. The vector traces from the map of the area (retrieved from OpenStreetMap) were reduced to the minimum amount needed to make the map recognizable and easily manageable by the embroidery PE-Design 10 software, which I used to transform the shapes into filling patterns.

Linen was chosen as the best material for the fabric base due to its availability, resistance and plain-aesthetic qualities. I decided to represent the covered areas we entered during the soundwalk as coloured reliefs completely made of grey/gold conductive thread. The park areas were left olive-green if not interactive and green mixed with the conductive thread if interactive. This was to allow the map to be clearly understood in its different elements. Courtyards we crossed were embroidered as flat areas in white with parts in conductive thread, whilst landmarks were represented with a mixture of pale grey, with conductive thread only on the side where the walk took place.

The River Thames, also present in the recordings, was depicted as a pale blue wavy surface with some conductive parts close to the sides where the walk took place. Buildings belonging to the area but not covered in the soundwalk were represented in flat pale grey hatch.

The engineering process

The fabric was meticulously embroidered with coloured rayon and conductive threads thanks to the precision of the embroidery machine. I tested the conductive thread and the different stitch configurations on a small sample of fabric to determine how well the capacitive charges and discharges caused by touching the conductive parts could be read by the breakout board.

The whole map consists of a graphical layer, an insulation layer, an embroidered circuit layer, a second insulation layer, and a bottom layer in neoprene which works as a soft base. Below the capacitive areas of the top layer I cut some holes in the insulation layer to allow the top layer to communicate with the circuit layer. Some of these areas have been also manually stitched to the circuit layer to keep the two layers in place. The fabric can be easily rolled and moved separately from the Bela board.

Some of the embroidered underlying traces. The first two traces appear too close in one point: when the fabric is not fully stretched they risk being triggered together!

Stitching the breakout board

Particular care was taken when connecting the circuit traces in the inner embroidered circuit layer to the capacitive pins of the breakout board. As this connection needs to be extremely solid it was decided to solder some conductive wire to the board, pass it through the holes beforehand, and then stitch the wires one by one to the correspondent conductive thread traces, which were previously embroidered.

Some pointers came from the process of working with the conductive thread:

  • Two traces should never be too close to one another or they will trigger false readings by shorting together.
  • A multimeter comes in handy to verify the continuity of the circuit. To avoid wasting time and material, it’s better to check for continuity on some samples before embroidering the final one as the particular materials and threads in use can behave very differently.
  • Be patient and carefully design your circuit according to the intended position of the capacitive boards. For example, I decided to place the two of them (to allow for 24 separate readings) in the top corners of the fabric.

Connecting with Bela:

The two breakout boards are connected through i2c to Bela which receives the readings from each pin of the breakout boards. The leftmost is connected through i2c to the other one, and this one goes to Bela. This cable is the only connection between the Fabric and Bela. It is possible to set an independent threshold for each pin, which will trigger the index releasing the correspondent recording. The code used to read the capacitive touch breakout board comes with the board and can be found here: examples/06-Sensors/capacitive-touch/.

MPR121 capacitive touch sensing breakout board connected to the i2c terminals of Bela.

The code to handle the recordings was nicely tweaked by Christian Heinrichs to add a natural fade in and fade out for the recordings. This code is based on the multi sample streamer example already available in Bela’s IDE which can be found here: examples/04-Audio/sample-streamer-multi/. Each recording has a pointer that keeps track of where the recording paused, so that touching the corresponding area again will resume playing from that point and not from the beginning. Multiple areas can be played at the same time allowing you to create experimental mixes of different ambiances.

Exhibition setting

This piece is best experienced through headphones as the recordings were made using binaural microphones. Nevertheless it is also possible to use speakers, with some loss of the spatial sonic image fidelity. In either case the audio output is taken directly from the Bela board. In the photograph below I made a wooden and perspex case for the board to protect it while it was installed in a gallery and powered the board with a USB 5V phone charger. Bela was set to run this project on start-up making it simple for gallery assistants to turn the piece on and off. The Aural Fabric is used for my PhD research, focused on novel approaches to strengthening the relationship between architecture and acoustics.  I’m engaging architecture students in sonic explorations and reflections on how architecture and its design contributes to defining our sonic environments.

Aural Fabric: Greenwich has been displayed at Sonic Environments in Brisbane among the installations and Inter/sections 2016 in London. More information documenting the making process is available here.

 

Sound Effects Taxonomy

At the upcoming International Conference on Digital Audio Effects, Dave Moffat will be presenting recent work on creating a sound effects taxonomy using unsupervised learning. The paper can be found here.

A taxonomy of sound effects is useful for a range of reasons. Sound designers often spend considerable time searching for sound effects. Classically, sound effects are arranged based on some key word tagging, and based on what caused the sound to be created – such as bacon cooking would have the name “BaconCook”, the tags “Bacon Cook, Sizzle, Open Pan, Food” and be placed in the category “cooking”. However, most sound designers know that the sound of frying bacon can sound very similar to the sound of rain (See this TED talk for more info), but rain is in an entirely different folder, in a different section of the SFx Library.

The approach, is to analyse the raw content of the audio files in the sound effects library, and allow a computer to determine which sounds are similar, based on the actual sonic content of the sound sample. As such, the sounds of rain and frying bacon will be placed much closer together, allowing a sound designer to quickly and easily find related sounds that relate to each other.

Here’s a figure from the paper, comparing the generated taxonomy to the original sound effect library classification scheme.

sfxtaxonomy

12th International Audio Mostly Conference, London 2017

by Rod Selfridge & David Moffat. Photos by Beici Liang.

Audio Mostly – Augmented and Participatory Sound and Music Experiences, was held at Queen Mary University of London between 23 – 26 August. The conference brought together a wide variety of audio and music designers, technologists, practitioners and enthusiasts from all over the world.

The opening day of the conference ran in parallel with the Web Audio Conference, also being held at Queen Mary, with sessions open to all delegates. The day opened with a joint Keynote from the computer scientist and author of the highly influential sound effect book – Designing Sound, Andy Farnell. Andy covered a number of topics and invited audience participation which grew into a discussion regarding intellectual property – the pros and cons if it was done away with.

Andy Farnell

The paper session then opened with an interesting talk by Luca Turchet from Queen Mary’s Centre for Digital Music. Luca presented his paper on The Hyper Mandolin, an augmented music instrument allowing real-time control of digital effects and sound generators. The session concluded with the second talk I’ve seen in as many months by Charles Martin. This time Charles presented Deep Models for Ensemble Touch-Screen Improvisation where an artificial neural network model has been used to implement a live performance and sniffed touch gestures of three virtual players.

In the afternoon, I got to present my paper, co-authored by David Moffat and Josh Reiss, on a Physically Derived Sound Synthesis Model of a Propeller. Here I continue the theme of my PhD by applying equations obtained through fluid dynamics research to generate authentic sound synthesis models.

Rod Selfridge

The final session of the day saw Geraint Wiggins, our former Head of School at EECS, Queen Mary, present Callum Goddard’s work on designing Computationally Creative Musical Performance Systems, looking at questions like what makes performance virtuosic and how this can be implemented using the Creative Systems Framework.

The oral sessions continued throughout Thursday, one presentation that I found interesting was by Anna Xambo titles Turn-Taking and Chatting in Collaborative Music Live Coding. In this research the authors explored collaborative music live coding using the live coding environment and pedagogical tool EarSketch, focusing on the benefits to both performance and education.

Thursday’s Keynote was by Goldsmith’s Rebecca Fiebrink, who was mentioned in a previous blog, discussing how machine learning can be used to support human creative experiences, aiding human designers for rapid prototyping and refinement of new interactions within sound and media.

Rebecca Fiebrink

The Gala Dinner and Boat Cruise was held on Thursday evening where all the delegates were taken on a boat up and down the Thames, seeing the sites and enjoying food and drink. Prizes were awarded and appreciation expressed to the excellent volunteers, technical teams, committee members and chairpersons who brought together the event.

Tower Bridge

A session on Sports Augmentation and Health / Safety Monitoring was held on Friday Morning which included a number of excellent talks. The presentation of the conference went to Tim Ryan who presented his paper on 2K-Reality: An Acoustic Sports Entertainment Augmentation for Pickup Basketball Play Spaces. Tim re-contextualises sounds appropriated from a National Basketball Association (NBA) video game to create interactive sonic experiences for players and spectators. I was lucky enough to have a play around with this system during a coffee break and can easily see how it could give an amazing experience for basketball enthusiasts, young and old, as well as drawing in a crowd to share.

Workshops ran on Friday afternoon. I went to Andy Farnell’s Zero to Hero Pure Data Workshop where participants managed to go from scratch to having a working bass drum, snare and high-hat synthesis models. Andy managed to illustrate how quickly these could be developed and included in a simple sequencer to give a basic drum machine.

Throughout the conference a number of fixed media, demos were available for delegates to view as well as poster sessions where authors presented their work.

Alessia Milo

Live music events were held on both Wednesday and Friday. A joint session titled Web Audio Mostly Concert was held on Wednesday which was a joint event for delegates of Audio Mostly and the Web Audio Conference. This included an augmented reality musical performance, a human-playable robotic zither, the Hyper Mandolin and DJs.

The Audio Mostly Concert on the Friday included a Transmusicking performance from a laptop orchestra from around the world, where 14 different performers collaborated online. The performance was curated by Anna Xambo. Alan Chamberlain and David De Roure performed The Gift of the Algorithm, which was a computer music performance inspired by Ada Lovelace. The wood and the water was an immersive performance of interactivity and gestural control of both a Harp and lighting for the performance, by Balandino Di Donato and Eleanor Turner. GrainField, by Benjamin Matuszewski and Norbert Schnell, was an interactive audio performance that demanded entire audience involvement, for the performance to exist, this collective improvisational piece demonstrated a how digital technology can really be used to augment the traditional musical experience. GrainField was awarded the prize for the best musical performance.

Adib Mehrabi

The final day of the conference was a full day’s workshop. I attended the one titled Designing Sounds in the Cloud. The morning was spent presenting two ongoing European Horizon 2020 projects, Audio Commons (www.audiocommons.org/) and Rapid-Mix. The Audio Commons initiative aims to promote the use of open audio content by providing a digital ecosystem that connects content providers and creative end users. The Rapid-Mix project focuses on multimodal and procedural interactions leveraging on rich sensing capabilities, machine learning and embodied ways to interact with sound.

Before lunch we took part in a sound walk around the Queen Mary Mile End Campus, with one of each group blindfolded, informing the other what they could hear. The afternoon session had teams of participants designing and prototyping new ways to use the APIs from each of the two Horizon 2020 projects – very much in the feel of a hackathon. We devised a system which captured expressive Italian hand gestures using the Leap Motion and classified them using machine learning techniques. Then in pure data each new classification triggered a sound effect taken from the Freesound website (part of the audio commons project). If time would have allowed the project would have been extended to have pure data link to the audio commons API and play sound effects straight from the web.

Overall, I found the conference informative, yet informal, enjoyable and inclusive. The social events were spectacular and ones that will be remembered by delegates for a long time.