Adaptive footstep sounds: new and improved for Unreal

From the thud clunk of an armored soldier in an action game to the creeping, ominous footsteps in a horror film, footstep sounds are one of the most widely sought after sound effects in creative content. But to get realistic variation, one needs hundreds of different samples for each character, each foot, each surface, and at different paces. Even then, repetition becomes a problem.

So at nemisindo.com , we developed a procedural model for generating footstep sounds without the use of recorded samples. We’ve recently improved it with higher sound quality, more features and an enhanced user experience. We just released this as the Nemisindo Adaptive Footsteps version 2.0 plug-in for game engines, available in the Unreal Marketplace. It offers a lot more than standard sample packs libraries: footsteps are generated in real-time, based on intuitive parameters that you can control.

The plugin provides benefits that no other audio plugin does;

• Customisable: 4 different shoe types, 7 surface types, and controls for randomisation, pace, step firmness, steadiness, etc. All of which can be controlled at runtime
• Convenient: Easy to set up, comes as both a plug-in or a custom node in MetaSounds, to help get started in no time.
• Improved: Version 2.0 features better sound quality, more features and better user experience.
• Lightweight: Uses very little less disk space; the entire code takes about the same space as one footstep sample.

In a research paper at the 152nd Audio Engineering Society Convention, we implemented a multilayer neural network architectures for footstep synthesis and compared the results with various sound synthesis methods, including Nemisindo’s online implementation. The neural approach is not yet applicable to most sound design problems, since it does not offer parametric control. But the listening test showed that Nemisindo’s procedural approach outperformed all other traditional sound synthesis approaches, and gave us insights that led to further improvements which we have put into this new version 2.0.

Here’s a short video introducing the new version:

So please check it out. Its a big footstep forward in procedural and adaptive sound design (sorry, couldn’t resist the wordplay 😁).

The State of the Art in Procedural Audio

I am pleased to announce that we have just published a large review article on procedural audio.

Procedural audio refers to the real-time generation of sounds that can adapt to changing input parameters. In its pure form, no recorded samples are stored, sounds are generated from ‘scratch’. This has huge benefits for game audio and VR, since very little memory is required, and sound generation can be controlled by the game or virtual environment. We have talked about procedural audio a few times before on this blog.

This review article is by Pedro Pestana of the Catholic University of Portugal, and Dimitris Menexopoulos and Josh Reiss, both from Queen Mary University of London and part of the research team behind this blog.

The article, published in the Journal of the Audio Engineering Society is quite a large one attempting to cover the state of the whole field. Its also a bit of an homage to all the great researchers that created it all.

Here’s the article, http://www.aes.org/e-lib/download.cfm/22346.pdf?ID=22346

With associated webpage https://dmenex.github.io/proceduralaudioreview/

And youtube video

Enjoy!

R&D audio software engineer position available

Nemisindo is seeking to employ an R&D software engineer to lead the technical work on the MetaSounds Plus project, enhancing the audio capability of Epic’s Unreal game engine. The position would involve using Unreal’s new MetaSounds format for developing procedural audio models (real-time sound synthesis) for the Unreal Engine, as well as improving and extending existing models. The ideal candidate should have a Master’s in a technical field with experience in C++ and audio programming. Experience in plug-in development, signal processing, machine learning and perceptual evaluation is also preferred. Knowledge of the game industry would also be beneficial.

The engineer would be encouraged to present their work at high profile events such as such as GameSoundCon, Game Developers Conference and Audio Engineering Society Conventions.

The position is for 1 year full-time (or longer if part-time) with a salary in the range of £38,000 to £42,000, depending on the skills and experience of the preferred candidate. The post is based at the Queen Mary Enterprise Zone in London, though remote working is possible. There is the possibility that this could lead to a long-term position, and potential for share options in the company.

If interested, please contact Nemisindo co-founder Josh Reiss, at josh@nemisindo.com / joshua.reiss@qmul.ac.uk . Please provide a CV with your email. We aim to interview shortlisted applicants around the end of July.

About the company: Nemisindo  (the Zulu word for ‘sound effects’) is a spin-out from Queen Mary University of London’s Centre for Digital Music. We offer sound design services based on innovative procedural audio technology. The following video gives an overview of Nemisindo, our technology and services https://youtu.be/j-pVTN5lhuM .

Our online service allows users to create sound effects for games, film and VR without the need for vast libraries of sounds. The platform provides a browser-based service with tools to create sounds from 75 classes (engines, footsteps, explosions…) and over 800 preselected settings (diesel generator engine, motorbike, fireworks, …). We also have procedural sound effects plug-ins for Unity and Unreal, and will soon release plug-ins for Wwise and VST too. In addition, we’ve received two MegaGrants from Epic Games, and have had contracts with The Key Clinic and Etobico.

Nemisindo awarded an EPIC MegaGrant for MetaSounds Plus

Nemisindo (from the Zulu word for sound effects) is a spin-out company from our research group, offering sound design services based around procedural audio technology. Nemisindo has been fairly quiet recently, developing new products and delivering some contract work. But expect to hear a lot from them in the near future, including this announcement.

A couple of years ago, Nemisindo was awarded a MegaGrant from Epic Games, to support their contributions to the Unreal community. It resulted in, among other things, delivery of a lot of procedural audio plug-ins for the Unreal game engine; the Action Pack, the Nature Pack, and Adaptive Footsteps.

Nemisindo is very pleased to announce that we have been awarded a new Epic MegaGrant, titled MetaSounds Plus. MetaSounds is the Unreal Engine’s new high-performance procedural audio system. It gives sound designers complete control over a Digital Signal Processing graph for the generation of sound sources. MetaSounds has the potential to be a game-changer in the industry. It is a fascinating technology that combines the best aspects of audio-oriented graphical programming languages (such as Max MSP and Pure Data), with the Unreal Engine workflow. It allows game developers to take their first steps in the world of procedural audio in an intuitive and seamless fashion.

The MetaSounds Plus project will greatly enhance Unreal Engine’s MetaSounds by delivering a suite of powerful nodes and assets. This benefits the Unreal community by providing developers with rich, intuitive tools to achieve great game sound design.

Here’s a video giving an overview of our proposal for the MetaSounds Plus project.

Nemisindo will soon be hiring an audio software engineer to support this project, so keep an eye out for more announcements. And keep an eye (and ear) out for some great new releases.

As always, if interested in knowing more, please get in touch.

Nemisindo releases procedural audio plugins for Unreal Engine 5.1

Nemisindo (the Zulu word for ‘sound effects’) is a spin-out company from our research group, offering sound design services based around procedural audio technology. Procedural audio refers to the real-time generation of sounds that can adapt to changing input parameters. In its pure form, no recorded samples are stored, sounds are generated from ‘scratch’. This has huge benefits for game audio and VR, since very little memory is required, and sound generation can be controlled by the game or virtual environment.

Nemisindo provide a free online sound design service, as well as procedural audio plug-ins for Unreal, Unity and other platforms.

A couple of months ago, Epic released Unreal Engine 5.1 , a major overhaul of their game and VR engine. We are pleased to announce that all Nemisindo’s Unreal procedural audio plugins are now available for UE 5.1. You can check them out from Nemisindo’s website, or by going to the Unreal Marketplace where you’ll find them as follows;

• Action Pack
• Adaptive footsteps
• Nature pack

And there are more procedural audio plugins in development from Nemisindo. Coming soon are;

• Engine / motor sounds for Unreal Engine
• Action Pack plugins for Wwise, the game audio engine
• Action Pack VST plugins for audio workstations

These have all been built and are in testing, revisions, packaging stages.

Funded PhD studentships in audio, games and more!

So its been a while since I’ve written a blog post. Life, work, and more work has made my time limited. But hopefully I’ll write more frequently in future.

The good news is that we have several fully funded PhD studentships which you or others you know might be interested in. They are all fully-funded, based at Queen Mary University of London, and starting September 2023.

As part of the AI and Music Doctoral (AIM) Training Centre,

• Auditory masking: advances and creative applications
• Machine learning of physical models – This is supported by the company Nemisindo.

As part of the Intelligent Games (IGGI) Doctoral Training Centre

• Machine learning of procedural audio  – This is supported by the company Nemisindo.

And for those students eligible for China Scholarship Council funding,

• Acoustic auto-focus – At the time of writing, the full topic description wasn’t available. But it should be there in a couple of days.

And finally, for those with UK residency status,

• Machine learning of physical models

With the exception of the last one, these are all 4 year studentships; the last one is for 3 years. Funding includes both a full tuition fee waiver and living stipend.

You’ll notice that there are related topics regarding machine learning applied to constructing simulations. This is an area where we have a strong interest, and there is scope for different projects pursuing different directions.

All the studentships are highly competitive, so its worth putting in a very strong application. And for those who fulfill CSC funding requirements or those who fulfill UK residency requirements, I recommend going for the last two, which may not be as competitive as the others. You can apply for more than one though.

If you or anyone you know is interested, please consider applying and feel free to ask me any questions. Good luck!

 

Follow The Sound: A Procedural Audio Minigame

In-game screenshot from Follow The Sound.

From the beginning of February until the end of April, IGGI PhD researcher Dimitris Menexopoulos did an internship at Nemisindo, a fast-growing start-up company that offers state of the art services in procedural audio. His main task was to design and build a minigame in Unreal Engine that would showcase as many of the procedural models featured in their plugins as possible.

As of early August 2022, Nemisindo has three plugins for the Unreal Engine out in the market (plus plugins for Unity, and more to come):

• The Action Pack, which includes Explosion, Gunshot, Rifle, Helicopter, Jet, Propeller, Rocket, Alarm, Alert, Siren and Fire models.
• The Adaptive Footsteps, which comes with 5 different shoe types, 7 surface types and additional controls for the pace, step firmness, steadiness, etc.
• The Nature Pack, which brings nature-inspired Droplets, Fire, Rain, Water, Waves and Wind models to any interactive environment.

As is illustrated above, the sonic options are many. But making a game that has a smooth audio-driven gameplay while concisely featuring many of the above models was a unique challenge.

The initial idea was to make a third person shooter kind of game, implementing the Rifle and Footsteps models on the main character and the rest in the scene’s environment and objects.

But that’s beaten to death, right? Plus, the intense action might shift the player’s attention from the audio to the graphics. So, how does one keep the game audio-focused?

After a bit further brainstorming, the answer came in the form of a common game mechanic that has stood the test of time like few, but approached with a slightly unorthodox mindset:

Collectibles. But emitting directional sound and being hidden in a 3D scene.

That’s roughly how Follow The Sound came to be. The objective is quite simple: there are ten sound emitting items scattered around the map. Each item is visually representative of the sound it emits. Can you locate and collect them all in under three minutes?

All sounds are procedurally generated at runtime.

Here’s some basic information about the sound models used in the game:

• Collectibles: Droplet, Gunshot, Rifle, Explosion, Jet, Helicopter, Propeller, Alarm, Fire, Rocket
• Footsteps: Trainers/Sneakers on Concrete, Gravel and Grass surfaces
• Nature Ambience: Wind model from the Nature Pack
• UI Collection Sound: Two-Tone Siren model (yes, with enough imagination, you can also make procedural UI sounds!)

Follow The Sound was created in Unreal Engine 4.26. All 3D models used in the minigame are free assets downloaded from the Unreal Marketplace (City Park Environment Collection by SilverTm) and from cgtrader.com.

Learn more about Nemisindo’s Action Pack, Adaptive Footsteps and Nature Pack at nemisindo.com.

Dimitris Menexopoulos would like to thank his PhD supervisor and CEO of Nemisindo, Dr. Joshua Reiss, for this creatively and technically didactic opportunity. Additionally, he’d like to thank Nemisindo members Selim Sheta, Jack Walters and Clifford Moses Manasseh for their valuable feedback and help in packaging the final version of the game.

You can freely download a Windows version of Follow The Sound here.

Engine sounds for game engines

Engines and motors feature heavily in video games, but they can be very problematic. As game car audio expert Greg Hill said, “For engine sounds most people think we just record a car and throw the sound into a folder and it magically just works in the game. Game sounds are interactive – unlike passive sound like movies where the sound is baked onto an audio track… sound has to be recorded a certain way so it can be decomposed and reconstructed into a ‘sonic model’ and linked to the physics so the player has full control over every parameter.”

And therein lies the issue; recorded sounds are fixed, but the game sounds need to be adaptable and controllable. One can get around this, but only with huge effort.

So Nemisindo (the Zulu word for Sound Effects, our start-up company) have brought their deep and advanced engine sound generator to Unity as an audio plugin.

The Nemisindo Engine implements a flexible, controllable and realistic engine for all your game vehicles. It is a Unity native audio plugin and can generate sound effects in real time, completely procedurally. The plugin offers an interactive interface with 14 parameters and 16 realistic presets (Formula One, monster truck, motorbike…). It also offers various functions to change the parameters of the sound effect at runtime. Hence the parameters can be linked to any in-game events which can lead to very organic and interesting results.

Here’s a demonstration video.

The engine is available as a native Unity audio plugin at https://assetstore.unity.com/packages/tools/audio/nemisindo-engine-procedural-sound-effects-222246

You can also try out the engine model on Nemisindo’s online sound design platform at

https://nemisindo.com/models/engine-advanced.html

Online sound design service – improved and relaunched

Last year, we launched an online sound design service at nemisindo.com . Nemisindo (the Zulu word for ‘sound effects’) is a start-up based on our procedural audio research at Queen Mary University of London.

Since then, we’ve launched procedural audio plug-ons for the Unreal and Unity game engines. We’ve also been continually improving the online service. But we’ve been holding back the big changes… until now.

Before, users could generate sounds from lots of different sound models, as well as access stored presets for those models. We’ve merged this all into one page, with easy and intuitive search features. And we’ve added more sounds (76 sound models with over 800 presets). So users don’t need to do much browsing around, they can easily find settings to generate the sorts of sounds they want.

We’re also really excited to be able to announce the launch of a new user community feature on our website. Now, users can create their own presets, save them to their profile, and view and use other peoples’ presets. This will enable a community to grow, building and sharing settings for generating great sound effects. And we benefit too, since the users will show us more interesting and creative ways of using our technology.

On top of all this, we’ve made a number of improvements across the website based on feedback from the community, including lots of helpful videos, simplifying interfaces and improvements to the underlying sound generation techniques.

Our online service is free to use, but please register on the site. And registration is needed to download sounds and store and share presets. As always, feel free to get in touch.

Thank you,
The Nemisindo Team

The sounds of nature, procedurally generated

#sounddesign #soundeffects #GameAudio

I was tempted to give this blog entry the name The Call of Nature, and then I remembered that for a lot of people, that has a very different meaning. 😉

Nature sounds are essential to sound design. They provide ambience and set the scene. Formally, they are almost all keynote sounds, that is, heard continuously or frequently enough to form the background against which other sounds are perceived.

But they can be very challenging to work with in creative sound design. You may have a hundred samples of ocean waves, but none of them matches the visuals. And this becomes even harder when the visual aspects might change depending on what happens in a game or VR context.

We want to help game developers implement dynamic and adaptive atmospheric sounds in their projects. So Nemisindo (the Zulu word for Sound Effects, our start-up company) has brought state-of-the-art procedural Nature models to Unreal Engine. From lonely humid caves, to huge hurricanes, we’ve got it covered.
The Nature Pack features 6 sound effects models for popular nature sounds:

• Droplets
• Wind
• Waves
• Rain
• Fire
• Water

These models generate audio in real time, completely procedurally so no samples are stored.

This approach offers a lot more than standard sample packs libraries. Sounds are generated in real-time, based on intuitive parameters that can be customized and linked to in-game events. The plugin is integrated with Unreal Engine, unlike other solutions that involve third party software. This means you can design your own sounds directly in the game or simulation editor.

With the Nature Pack, you’ll be able to create incredibly detailed interactive audio scenes, with roaring flames, crashing waterfalls and gentle waves breaking on the shore. The plugin comes with 50+ presets so you can get started in no time. And since it does not rely on pre-recorded samples, it’s very lightweight compared to sample libraries.

The Nature Pack plugin is available in the Unreal Marketplace at unrealengine.com/marketplace/product/nemisindo-nature-pack

And here’s a short video introducing the plugin and its features: