making soundfont babies?
I wanted to take the same note from three instruments and make the sound in between them. This is not a crossfade. The attack, the formants and the harmonics of the new note should each sit somewhere between its parents.
I made this for my synthetic data pipeline. The pipeline renders MIDI through sample libraries to make training audio for transcription models. The problem is that there are only so many distinct instruments to render with, so a model may learn those exact sounds instead of the instrument in general. If I could make new instruments in between the ones I have, I would get many more timbres without recording anything.
the demo
The demo uses six instruments from the Sonatina Symphonic Orchestra (harp, piano, flute, violin, viola and oboe), with one note each at E5. Three of them sit on the dots. Click a dot to swap its instrument. The pink dot is the playhead. Wherever you drag it, the note Mozart is playing is rebuilt from the three instruments at the weights the lines show. You can drag the instruments too.
If you put the playhead on one instrument, you will hear exactly that sample. If you slide it toward another one, the breath noise, the vibrato and the length of the attack all move over gradually. One instrument does not fade out under the other. Try putting the harp or the piano on a dot and moving toward it. The note gets shorter as you get closer, because the decay is interpolated too, not only the timbre.
how it works
Every note is taken apart in the MCLT domain into four parts. The first is a level per frame. The second is a spectral envelope (Röbel and Rodet's true envelope, which is a cepstral smoothing that is iterated so it rides over the harmonic peaks). The third is the fine structure left under the envelope. The fourth is time. The sources are aligned to each other with dynamic time warping on level and low-order cepstrum. This is so that a slow attack and a fast attack interpolate in length instead of smearing.
Then the weights average each part in the domain where averaging sounds right. The level is averaged in dB, the envelope in dB, and the fine structure in a loudness-like |a|^0.6. That model says what the morph should sound like, but the waveform comes from the recordings themselves. Each source is warped onto the shared timeline in the time domain with WSOLA, so the phases stay intact. The warped sources are summed with the weights. The mix keeps its phases, and its magnitudes are swapped for the model's magnitudes. Two passes of Griffin-Lim make the two agree. The phase always comes from a mix of the recorded sources, so an attack will still sound like an attack.
The full version morphs the broad formant band as an optimal transport barycentre on a log-frequency axis. That way a formant at 500 Hz in one flute and 1 kHz in another lands near 700 Hz, instead of turning into two bumps at half the height. The browser build uses plain dB interpolation of the envelope (the env_mode="log" setting in Python). Between instruments as different as a harp and an oboe, that does cost some quality. It is the first thing I would bring over to the browser.
in the browser
The numpy version renders a weight vector in about 130 ms on a desktop CPU. The site version is a line-for-line port to TypeScript. The DCT-IV and the cepstral DCTs go through a radix-2 FFT, and the time warp is the same WSOLA search. It runs in a Web Worker so that the page does not freeze. The three sources are analysed once when you load them. After that, each drag of the dot is one render. Notes that are already playing keep their old sample, and new notes use the new one. The status line under the demo shows the numbers from your machine.
I checked the port against the Python build. For ten random six-way weight vectors, the waveforms from the two versions correlate at 0.99 or higher. A one-hot weight gives back the source, bit for bit. The code is in the site's repository under lib/babymaker.