The sounds of nature, procedurally generated

#sounddesign #soundeffects #GameAudio

I was tempted to give this blog entry the name The Call of Nature, and then I remembered that for a lot of people, that has a very different meaning. 😉

Nature sounds are essential to sound design. They provide ambience and set the scene. Formally, they are almost all keynote sounds, that is, heard continuously or frequently enough to form the background against which other sounds are perceived.

But they can be very challenging to work with in creative sound design. You may have a hundred samples of ocean waves, but none of them matches the visuals. And this becomes even harder when the visual aspects might change depending on what happens in a game or VR context.

We want to help game developers implement dynamic and adaptive atmospheric sounds in their projects. So Nemisindo (the Zulu word for Sound Effects, our start-up company) has brought state-of-the-art procedural Nature models to Unreal Engine. From lonely humid caves, to huge hurricanes, we’ve got it covered.
The Nature Pack features 6 sound effects models for popular nature sounds:

• Droplets
• Wind
• Waves
• Rain
• Fire
• Water

These models generate audio in real time, completely procedurally so no samples are stored.

This approach offers a lot more than standard sample packs libraries. Sounds are generated in real-time, based on intuitive parameters that can be customized and linked to in-game events. The plugin is integrated with Unreal Engine, unlike other solutions that involve third party software. This means you can design your own sounds directly in the game or simulation editor.

With the Nature Pack, you’ll be able to create incredibly detailed interactive audio scenes, with roaring flames, crashing waterfalls and gentle waves breaking on the shore. The plugin comes with 50+ presets so you can get started in no time. And since it does not rely on pre-recorded samples, it’s very lightweight compared to sample libraries.

The Nature Pack plugin is available in the Unreal Marketplace at unrealengine.com/marketplace/product/nemisindo-nature-pack

And here’s a short video introducing the plugin and its features:

Working with the Web Audio API out now- book, source code & videos

I was tempted to call this “What I did during lockdown”. Like many people, when Covid hit, I gave myself a project while working from home and isolated.

The Web Audio API provides a powerful and versatile system for controlling audio on the Web. It allows developers to generate sounds, select sources, add effects, create visualizations and render audio scenes in an immersive environment.

In recent years I had developed a Love / Mild Annoyance (hate is too strong) relationship with the Web Audio API. And I also noticed that there really wasn’t a comprehensive and useful guide to the Web Audio API. Sure, there’s the online specification and plenty of other documentation, but there’s a whole lot that it leaves out, whether one wants to do FIR filtering or one wants the easiest way to record the output of an audio node, for instance. And there’s nothing like a teaching book or a reference that one might keep handy while doing Web Audio programming.

So writing that book, titled Working with the Web Audio API, became my Covid project, and its finally hit the shelves! Its part of the AES Presents book series, and the publisher’s link to the book is;

https://www.routledge.com/Working-with-the-Web-Audio-API/Reiss/p/book/9781032118673

but you can find it through all the usual places to buy books. The accompanying source code is available at;

https://github.com/joshreiss/Working-with-the-Web-Audio-API

I think there’s about 70 source code examples, covering every audio node and every important feature of the API.

And I’ve made about 20 instructional videos covering many aspects in the YouTube channel.

https://tinyurl.com/y3mtauav

I’ll keep improving the github repository and YouTube channel whenever I get a chance. So please, check it out and let me know what you think. 🙂

Adaptive footstep sound effects

Adaptive footsteps plug-in released for Unreal and Unity game engines

From the creeping, ominous footsteps in a horror film to the thud clunk of an armored soldier in an action game, footstep sounds are one of the most widely souht after sound effects in creative content. But to get realistic variation, one needs hundreds of different samples for each character, each foot, each surface, and at different paces. Even then, repetition becomes a problem.

So at nemisindo.com , we’ve developed a procedural model for generating footstep sounds without the use of recorded samples. We’ve released it as the Nemisindo Adaptive Footsteps plug-in for game engines, available in the Unity Asset Store and in the Unreal Marketplace. You can also try it out at https://nemisindo.com/models/footsteps.html . It offers a lot more than standard sample packs libraries: footsteps are generated in real-time, based on intuitive parameters that you can control.

The plugin provides benefits that no other audio plugin does;

• Customisable: 4 different shoe types, 7 surface types, and controls for pace, step firmness, steadiness, etc.
• Convenient: Easy to set up, comes with 12 presets to get started in no time.
• Versatile: Automatic and Manual modes can be added to any element in a game scene.
• Lightweight: Uses very little less disk space; the entire code takes about the same space as one footstep sample.

In a research paper soon to appear at the 152nd Audio Engineering Society Convention, we tried a different approach. We implemented multilayer neural network architectures for footstep synthesis and compared the results with real recordings and various sound synthesis methods, including Nemisindo’s online implementation. The neural approach is not yet applicable to most sound design problems, since it does not offer parametric control. But the listening test was very useful. It showed that Nemisindo’s procedural approach outperformed all other traditional sound synthesis approaches, and gave us insights that led to further improvements.

Here’s a short video introducing the Unity plugin:

And a video introducing it for Unreal

And a nice tutorial video on how to use it in Unreal

So please check it out. Its a big footstep forward in procedural and adaptive sound design (sorry, couldn’t resist the wordplay 😁).

Lights, camera… Action sound effects pack!

Procedural audio refers to the real-time generation of sounds that can adapt to changing input parameters. In its pure form, no recorded samples are stored, sounds are generated from ‘scratch’. This has huge benefits for game audio and VR, since very little memory is required, and sound generation can be controlled by the game or virtual environment.
Nemisindo (the Zulu word for ‘sound effects’) is a spin-out company from our research group, offering sound design services based around procedural audio technology. Previously, we blogged about Nemisindo’s first procedural audio plugin for the Unreal game engine, the Action Pack. Since then we have released an adaptive footsteps plugin for Unreal too. But today I’ll give you more detail on the Action Pack, now available for both Unreal and Unity.

Here’s a video all about the Action Pack for Unity;

The Action Pack contains 11 different procedural audio models, all based atleast in part from research in this group and within Nemisindo. Here’s the list, with links to online demonstrators of each sound model;

• Explosion – a wide ranging sound model, capable of generating sounds from fireworks to thuds to grenades to bombs and more
• Fire – the sound of a fire is a well-known sound texture, necessary for the ambience of many scenes. To get it right, we offer control of all the key characteristics
• Gunshot – capturing characteristics of all the elements of a gunshot; the shell, the gassing, the bullet casing
• Rifle – aimed at precise models of particular rifle designs, like the Winchester or Beretta
• Helicopter – generates the sounds of a helicopter (engine, blades…), for arbitrary speeds, listener positions and more
• Propeller – based on our research into aeroacoustics, this models the sound of aircraft propellers, from modern small drones to the World War 2 bomber the Flying Fortress
• Jet – a powerful jet engine, with subtle control over things like thrust and turbine whine
• Rocket – capturing the intense, powerful sound of a rocket launch
• Alarm – based on the same principles for the design of car alarms, fire alarms, motion sensors…
• Alert – red alert, background emergency sounds and more
• Siren – did you know that police and ambulance sirens have different sounds in different states and countries,, with different settings for different situations? This sound model matches the range of siren sounds in at least five countries.

The biggest benefit of using these plug-ins, in my opinion, is how easy the sound design becomes. What would have been hours or even days of sourcing sound samples, processing them, loading them and assigning them to game events, now becomes just minutes to link plug-in parameters to game events.

You can get the Action Pack in the Unity Asset Store and in the Unreal Marketplace .

Exploiting Game Graphics Rendering For Sound Generation

A fascinating game audio research topic

When rendering the assets available in them, video games are often pushing the processing limits of their host machines, be it personal computer systems, dedicated gaming consoles or even cell phone devices. As modern video games continue to rapidly develop into increasingly rich and varied experiences, both aesthetic and technical considerations, among other factors, become important components to their success.

Particularly the graphics department is the one that often makes or breaks a game from the get-go for many players and this is why developers pay extra attention to it. One of the techniques that are commonly used in order to provide variety and realism in game graphics is procedural generation, sometimes also called procedural content generation (PCG in short). The term can be defined in different ways, but I believe the definition from Wikipedia is quite accurately descriptive:

In computing, procedural generation is a method of creating data algorithmically as opposed to manually, typically through a combination of human-generated assets and algorithms coupled with computer-generated randomness and processing power.

_{No Man’s Sky Next Generation (2020) by Hello Games features almost exclusively PCG assets, including audio.}

Despite the fact that game graphics frequently enjoy the affordances of PCG techniques, game audio has only in the last decade, or so, begun to take advantage of it. Why? In his book Designing Sound (2010), procedural audio expert Andy Farnell gives the following explanation:

In fact, game audio started using procedural technology. Early game consoles and personal computers had synthesiser chips that produced sound effects and music in real time, but once sample technology matured it quickly took over because of its perceived realism. Thus synthesised sound was relegated to the scrapheap.

The complexity of modern games has come to challenge this approach. Indeed, samples are frozen audio snapshots in time and creative processing can only alter them so far in providing convincing realism, depth and variety at the end of the day. Procedural audio on the other hand, does exactly that by definition and the current state of technology is developed enough to support its successful implementation. Of course, different game development scenarios call for different solutions and more often than not, a mixed approach is preferable.

To get a taste of cutting-edge procedural audio effects implementation, below you can find an example using the latest plugin for the Unreal Engine by Nemisindo, which was released a few days ago.

_{All sounds are procedurally generated in real-time during gameplay.}

So, far we took a look at procedurally generated graphics independently from audio. But how can existing graphics information, available in the game engine, be used to generate the sounds and music produced when objects interact in a game environment?

At this point, it is important to clarify the distinction between data and metadata inside a game engine. From the perspective of graphics, data can be perceived as everything we see, such as the shapes of the 2D or 3D models, the textures, the colours etc. Conversely, audio data are anything we hear, such as the music, sound effects and voiceovers. Metadata in both these cases, are all the stored parameters in memory that carry information about the state of each data asset at a certain point in time.

A really good way to start realizing graphics-based procedural audio is to try to translate and share metadata among PCG algorithms for audio and graphics. What if the varying state of a moving character’s speed could be directly used to produce infinitely unique step variations? What if different shoe or ground materials could drive different parameters of step sound physical models at runtime? What if information about the weather and time of the day could trigger different interactive music cues and set specific audio effect parameters to create unique soundscapes at every moment of gameplay? The possibilities are endless and go beyond games, as game engines are steadily finding their place in other fields, ranging from multimedia art to computational medicine.

Thank you for reaching this far, we hope you found this article informative. We will leave you with a video demonstration of Homo Informis, a music creation system by new IGGI PhD reseracher Dimitris Menexopoulos, which works by translating still and moving images into sound and by exposing the sonic parameters to be controlled by body gestures and EEG waves in real time.

_{Dimitris Menexopoulos — Homo Informis (2020)}

Nemisindo launches procedural audio plugins for Unreal Engine

Nemisindo is a spin-out company from our research group, offering sound design services based around procedural audio technology. Back in August we blogged about the launch of Nemisindo’s online service at https://nemisindo.com . Now, Nemisindo has a new launch, targeted specifically at game developers.

The Nemisindo team is pleased to introduce fully procedural audio plugins for Unreal Engine: the Nemisindo Action Pack. Nemisindo have brought our state-of-the-art models to Epic Games’ renowned game engine, enabling true procedural sound effects generation within Unreal projects.

Procedural audio refers to the real-time synthesis of sounds depending on specific input parameters. Much like how a landscape can be procedurally generated based on certain inputs, like “elevation”, “variation”, or “biome type”, a helicopter sound can be procedurally generated based on parameters like “rotor speed”, “engine volume”, or “blade length”. Procedural audio is the next generation of sound technology that creates realistic immersive soundscapes that are fully interactive, adaptive and dynamic.

The Nemisindo Action Pack includes 11 different sound classes: Explosion, Gunshot, Rifle, Helicopter, Jet, Propeller, Rocket, Alarm, Alert, Siren, and Fire. Each sound class can generate audio in real-time, and, for comes with built-in presets for popular settings (such as ‘bomb’ and ‘thud’ for the Explosion model, or ‘emergency vehicle horn’ for the Siren model). The Nemisondo Action Pack plugin enables Unreal developers to:

• Design sound exactly how you want it – directly inside the Unreal Editor
• Link model parameters to in-game events via Blueprints
• Add any model to any actor, instantly turning the actor  into a sound generator
• Easily implement adaptive audio without external software dependencies
• Reduce disk space with zero-reliance on sound samples

The Nemisindo Action Pack is available in the Unreal Marketplace at: 

https://www.unrealengine.com/marketplace/product/nemisindo-action-pack .

And here’s a short video introducing the Action Pack and its features:

There’s another great video about it by The Sound Effects Guy here (we don’t know him personally and didn’t pay him for it or anything like that),

Nemisindo’s mission is to generate any sound effect procedurally, doing away with the need for sample libraries. Nemisindo’s technology can generate entire auditory worlds, create soundscapes based on sounds that have never been heard before, or enable every object in a VR/AR world to generate sounds that can adapt to changing conditions. These sound effects can also be shaped and crafted at the point of creation rather than via post-processing, breaking through the limitations of sampled sounds.

Named after the Zulu for “sounds/noise”, Nemisindo is an Epic MegaGrant recipient, awarded to support their contribution to procedural audio in the Unreal community.