Congratulations, Dr. Marco Martinez Ramirez

Today one of our PhD student researchers, Marco Martinez Ramirez, successfully defended his PhD. The form of these exams, or vivas, varies from country to country, and even institution to institution, which we discussed previously. Here, its pretty gruelling; behind closed doors, with two expert examiners probing every aspect of the PhD. And it was made even more challenging since it was all online due to the virus situation.
Marco’s PhD was on ‘Deep learning for audio effects modeling.’

Audio effects modeling is the process of emulating an audio effect unit and seeks to recreate the sound, behaviour and main perceptual features of an analog reference device. Both digital and analog audio effect units  transform characteristics of the sound source. These transformations can be linear or nonlinear, time-invariant or time-varying and with short-term and long-term memory. Most typical audio effect transformations are based on dynamics, such as compression; tone such as distortion; frequency such as equalization; and time such as artificial reverberation or modulation based audio effects.

Simulation of audio processors is normally done by designing mathematical models of these systems. Its very difficult because it seeks to accurately model all components within the effect unit, which usually contains mechanical elements together with nonlinear and time-varying analog electronics. Most audio effects models are either simplified or optimized for a specific circuit or  effect and cannot be efficiently translated to other effects.

Marco’s thesis explored deep learning architectures for audio processing in the context of audio effects modelling. He investigated deep neural networks as black-box modelling strategies to solve this task, i.e. by using only input-output measurements. He proposed several different DSP-informed deep learning models to emulate each type of audio effect transformations.

Marco then explored the performance of these models when modeling various analog audio effects, and analyzed how the given tasks are accomplished and what the models are actually learning. He investigated virtual analog models of nonlinear effects, such as a tube preamplifier; nonlinear effects with memory, such as a transistor-based limiter; and electromechanical nonlinear time-varying effects, such as a Leslie speaker cabinet and plate and spring reverberators.

Marco showed that the proposed deep learning architectures represent an improvement of the state-of-the-art in black-box modeling of audio effects and the respective directions of future work are given.

His research also led to a new start-up company, TONZ, which build on his machine learning techniques to provide new audio processing interactions for the next generation of musicians and music makers.

Here’s a list of some of Marco’s papers that relate to his PhD research while a member of the Intelligent Sound Engineering team.

• M. A. Martinez Ramirez, E. Benetos, J. D. Reiss, time-varying and nonlinear audio processing using deep neural networks, patent application number PCT/GB2020/051150, 2020
• M. A. Martinez Ramirez, E. Benetos, and J. D.Reiss, J. D., Deep learning for black-box modeling of audio effects. Applied Sciences, 10(2), 638, 2020
• M. A. Martinez Ramirez, E. Benetos, and J. D. Reiss, ‘Modeling plate and spring reverberation using a DSP-informed deep neural network,’ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020
• M. A. Martinez Ramirez, E. Benetos, and J. D. Reiss, “A general-purpose deep learning approach to model time-varying audio effects,” Digital Audio Effects Conference (DAFx), 2019
• M. A. Martinez Ramirez and J. D. Reiss, ‘Modeling Nonlinear Audio Effects with End-to-end Deep Neural Networks,’ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019
• M. A. Martinez Ramirez and J. D. Reiss, ‘End-to-end equalization with convolutional neural networks,’ Digital Audio Effects (DAFx), Aveiro, Portugal, Sept. 4–8 2018.
• M. A. Martinez Ramirez and J. D. Reiss, ‘Stem Audio Mixing as a Content-Based Transformation of Audio Features,’ IEEE 19th International Workshop on Multimedia Signal Processing, Luton, UK, Oct. 16-18, 2017.
• M. A. Martinez Ramirez and J. D. Reiss, ‘Analysis and Prediction of the Audio Feature Space when Mixing Raw Recordings into Individual Stems,’ 143rd AES Convention, New York, Oct. 18-21, 2017.
• M. A. Martinez Ramirez and J. D. Reiss, ‘Deep Learning and Intelligent Audio Mixing,’ 3rd Workshop on Intelligent Music Production, Salford, UK, 15 September 2017.
• M. A. Martinez and J. D. Reiss, “Intelligent audio mixing using deep learning“, DMRN+11: Digital Music Research Network Workshop, Dec. 2016

Congratulations again, Marco!

Congratulations, Dr. Will Wilkinson

This afternoon one of our PhD student researchers, Will Wilkinson, successfully defended his PhD. The form of these exams, or vivas, varies from country to country, and even institution to institution, which we discussed previously. Here, its pretty gruelling; behind closed doors, with two expert examiners probing every aspect of the PhD.
Will’s PhD was on ‘Gaussian Process Modelling for Audio Signals.’

Audio signals are characterised and perceived based on how their spectral make-up changes with time. Latent force modelling assumes these characteristics come about as a result of a common input function passing through some input-output process. Uncovering the behaviour of these hidden spectral components is at the heart of many applications involving sound, but is an extremely difficult task given the infinite number of ways any signal can be decomposed.

Will’s thesis studies the application of Gaussian processes to audio, which offer a way to specify probabilities for these functions whilst encoding prior knowledge about sound, the way it behaves, and the way it is perceived. Will advanced the theory considerably, and tested his approach for applications in sound synthesis, denoising and source separation tasks, among others.

http://c4dm.eecs.qmul.ac.uk/audioengineering/latent-force-synthesis/ – demonstrates some of his research applied to sound synthesis, and https://fxive.com/app/main-panel/Mammals.html is a real-time demonstration of his Masters work on sound synthesis for mammalian vocalisations.

Here’s a list of all Will’s papers while a member of the Intelligent Sound Engineering team and the Machine Listening Lab.

• W. J. Wilkinson, M. R. Andersen, J. D. Reiss, D. Stowell, A. Solin, ‘Unifying probabilistic models for time-frequency analysis,’ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 20196.
• W. J. Wilkinson, J. D. Reiss, D. Stowell, ‘A Generative Model for Natural Sounds Based on Latent Force Modelling,’ Arxiv pre-print version. International Conference on Latent Variable Analysis and Signal Separation, Guildford, UK, July 2018.
• W. J. Wilkinson, J. D. Reiss and D. Stowell, ‘Latent Force Models for Sound: Learning Modal Synthesis Parameters and Excitation Functions from Audio Recordings,’ 20th International Conference on Digital Audio Effects (DAFx-17), Edinburgh, UK, September 5–9, 2017
• W. Wilkinson, J. D. Reiss, A Synthesis Model for Mammalian Vocalization Sound Effects, 61st International AES Conference: Audio for Games, London 2016
• W. Wilkinson, D. Stowell and J. D. Reiss, “Performable spectral synthesis via low-dimensional modelling and control mapping“, DMRN+11: Digital Music Research Network Workshop, Dec. 2016
• W. J. Wilkinson, Michael Riis Andersen, Joshua D. Reiss, Dan Stowell, and Arno Solin, “End-to-End Probabilistic Inference for Non-stationary Audio Analysis” Int. Conference on Machine Learning (ICML), 2019

What a PhD thesis is really about… really!

I was recently pointed to a blog post about doing a PhD. It had lots of interesting advice, mainly along the lines of ‘if you are finding it difficult, don’t worry, that probably means you’re doing it right.’ True, and good advice to keep in mind for PhD researchers who might be feeling lost in the wilderness. But it reminded me that I’d recently given a talk about PhD research, based on experience I have either examining or supervising dozens of theses, and some of the main points that I made are worth sharing. And I think they are applicable to research-based PhDs across lots of different disciplines.

First off, lets think of a few things that a PhD thesis is not supposed to be;

• A thesis isn’t easy

See the blog I mentioned above. Easy research may still be publishable, but its not going to make a thesis. If you’re finding it easy, you’re probably missing the point.

• A thesis is not only what you already know

I’ve known researchers unwilling to learn a bit of new maths, or learn what’s going on under the hood in the software they use. Expect the research to lead you out of your comfort zone.

• A thesis isn’t just something you do to get a phd

It’s not simply a box that needs to be checked off so that you can get ‘Doctor’ next to your name.

• A thesis isn’t obvious

If you and most others can predict the outcome in advance based on common sense, then why do it?

• A thesis isn’t just several years of hard work

It may take years of hard work to achieve, but that’s not the point. You don’t get a PhD just for time and effort.

• A thesis isn’t about building a system

that’s challenging and technical, and may be a byproduct of the research, but its not the research result.

• A thesis isn’t a lot of little achievements

I’ve seen theses that read a bit like ‘I did this little interesting thing, then this other one, then another one…’ That doesn’t look good. If no one contribution is strong enough to be a thesis, then just putting them all into one document still isn’t a strong contribution. Note that in some cases, you can do a ‘thesis by publication’, which is a collection of papers, usually with an introduction and some wrapper information. But in which case it should still tie together with an overall contribution.

So with that in mind, lets now think about what a thesis is, with a few highlighted aspects that are often neglected.

• A thesis advances knowledge

That’s the key. Some new understanding, new insights, backed up by evidence and critical thinking. But this also suggests that it needs to actually be an advance, so you really need to know the prior art. How much reading of the literature one should do is a different question, and depends on the topic, the field, and the researcher. But in my experience, researchers generally don’t explore the literature deep enough. One thing is for sure though; if the researcher ever makes the claim that no one has done this before, they better have the evidence to back that up.

• A thesis is an argument

The word ‘thesis’ comes from Greek, and means an argument in the sense of putting forth a position. That means that there needs to be some element of controversy in the topic, and the thesis provides strong evidence supporting a particular position. That is, someone knowledgeable in the field could read the abstract and think, ‘no, I don’t believe it,’ but then change his or her mind after reading the whole thesis.

• A thesis tells a story

People tend to forget that it’s a book. Its meant to be read, and in some sense, enjoyed. So the researcher should think about the reader. I don’t mean it should be silly or salacious, but it should be engaging, and the researcher should always consider whether they (or at least some people in the field) would want to read what they’d written.