Introduction

• WebAssembly deliver performance for rendering realtime audio
• Low latency possible with AudioWorklet
• Let's create a synthesizer and sequencer in WebAssembly

Background

• Trackers and m68k assembly on Commodore Amiga in late 80s/90s
• Created a DAW for Java in 2005 ( Frinika )
• Recent years wanted to express music in code with javascript controlling a midi synth
• Was introduced to 4klang (x86 assembly synth) which is capable of rendering fantastic music from just a few kb
• Could something similar to 4klang be done in WebAssembly?

Warning

• Experimenting with synthesizers can produce sudden unexpected very loud noise, which may damage your hearing
• Keep the volume low, especially if using headphones
• Make sure you know where the mute button is :-)

The basics

• The simplest instrument
• Add to the mix

Sequencer

(same as in 4klang)

• A simple pattern sequencer
• Short fixed length patterns
  

  
                  [0, 0, 0, 0, 0, 0, 0, 0], 
                  [64, 0, 65, 0, 0, 67, 64, 0], 
                  [22, 23, 34, 34, 34, 0, 44, 45],
                  [22, 33, 0, 0, 34, 0, 44, 55]
          

• Track for each instrument with a list of patterns to play
  

  
                  [0, 1, 0, 2],
                  [1, 1, 3, 3],
                  [0, 0, 1, 2],
          

• this is all it takes to orchestrate the instruments

Generate sequencer data from code

javascript api to create patterns addInstrument("bass", { type: 'note' }); addInstrument("lead", { type: 'note' }); playPatterns({ "bass": pp(4,[g1,,g2]), "lead": pp(4, [c2, c5, c5(1)]) }); playPatterns({ "bass": pp(4,[g1,,g2]), "lead": pp(4, [c2, c5, c6(1)]) }); Pattern data

Patterns to be played per instrument

Record MIDI and generate code

• While playing, midi input data is stored to patterns
• If we want to use the recording, we can paste it as code
• Pattern data is "reverse engineered" to javascript code, with durations on the notes instead of repeated hold commands

AssemblyScript

why did I choose it?

• High level readability
• Low level control
• Pure WebAssembly output (no additional js lib)
• Builds optimized for speed and size ( Binaryen )
• Create WebAssembly binaries in the browser
• Great for live coding: rapid development, instant results, directly in the browser!

Synthesizing instruments in AssemblyScript

No sampled data, just code

Data driven or code driven?

• You can synthesize an instrument by connecting envelopes, oscillators, filters etc.
• Typical to create a data structure to be interpreted at runtime
• With AssemblyScript/WebAssembly we can instead generate and compile the code in the browser
• Just like modern web-frameworks resolving configuration at compile-time (such as language)
• Our binary can contain the logic directly rather than an interpreter of data describing the logic
• Faster and smaller builds, no interpreter overhead
• Not just for synthesizers but also for e.g. smart contracts
• Compiling is cheap, make pre-configured binaries rather than configuring at runtime

AudioWorklet

• The "proper" way of using AudioWorklet would be to have one node per instrument and let WebAudio do the orchestration/mixing
• But then we couldn't have music produced by a single WebAssembly executable binary
• Made a polyfill for the purpose of serving this app
• AudioWorklet model of render audio callback for WASI? ( similar to JACK and Core Audio )

Latency

• Realtime rendering with low latency only possible with AudioWorklet
• Can still experience clicks and glitches on the lowest latency
• Possible to request higher latency when creating the AudioContext
• "interactive" or "playback" or custom latency
• Also helps enabling "Use realtime priority thread for Audio Worklet" flag in chrome

Export WASM

create self-contained WebAssembly executables that can render music

• Use Wasmer to export raw audio data and import into e.g. Audacity with 32-bit float, little endian, stereo:
  

  wasmer song.wasm > song.raw

• Play directly from Wasmer by piping to sox:
  

  wasmer song.wasm | sox -S -t raw -b 32 -e float -r 44100 -c 2 - -d

• Create wav directly from Wasmer by piping to sox:
  

  wasmer song.wasm | sox -S -t raw -b 32 -e float -r 44100 -c 2 - song.wav

• or use the exported WASM binary from nodejs or the browser