Audio-reactive motion

Any property you can animate can also follow the sound a timeline plays. A logo can thump with the kick, a bar can rise with one band of the spectrum, a colour can flash on the beat, and a shader can swell with the bass. PopCraft listens to the audio itself, frame by frame. What you see while scrubbing is exactly what the exported video shows.

How to

Make a layer pulse with the bass

1. Put some sound on the timeline: drop a music file on the frame's audio lanes, or on a composition's. Or start from one of the music-video templates.
2. Select the layer, twirl its row open on the timeline, and pick the property (Scale X, say).
3. Click the row's Behaviours button, then + and Audio: bass kick.
4. Set To to how far it should go at full bass (0.2 for scale). Press Space.

The card's Now bar shows how hard the sound drives the behaviour at the playhead. If it stays at 0, the timeline has no sound yet, or Gain is too low.

Make bars dance to the spectrum

Add Audio: frequency band to each bar's Scale Y. Set its Band from 0 on the first bar (40 Hz) to 0.9 on the last (about 10 kHz), set Applies to Replace the value, and give it a fast Attack and a slower Release. Every bar then rises fast and falls back slowly, the way an equalizer does. The Spectrum bars component in the Audio reactive library is exactly this.

Flash a colour on the beat

On a colour track (a fill's Color), add Audio: beat flash and pick the colour it flashes to under Flash to.

Add a visualizer

1. 1.Select the background and open its fill0:00
2. 2.Pick a visualizer under Audio visualizers0:01
3. 3.Press Space: it follows the music0:03

Description

The Lo-fi loop visualizer in Animate, its background layer (Liquid ink) selected. In the Design panel its fill opens; under Audio visualizers Kaleidoscope is picked and the background becomes a turning kaleidoscope. Space plays the track: the pattern zooms with the mids and brightens with the level.

1. Select a rectangle (or the frame's background) and open its fill.
2. Choose Shader, then one of the Audio visualizers.
3. Its audio inputs follow the music right away: each is a shader param with an Audio behaviour on it. Tune the look with the other params, or change a behaviour's signal and gain like any other.

Where the motion comes from

Nothing is keyed to a song. When a clip lands on the timeline, PopCraft decodes the file and analyses it in the background, once per file: its loudness, bass, mids, treble, spectrum, onsets and beats, 100 times a second. Each frame then reads that analysis at the moment being played and runs it through the behaviours.

• Replace the track and everything that follows it follows the new one, as soon as the new file is analysed.
• A clip's volume, fades and speed count. A fading clip drives less, and a muted clip drives nothing. With several clips playing at once, each behaviour follows the loudest.
• It isn't a live input. PopCraft doesn't listen to a microphone or to whatever else the computer is playing. Because the file is analysed ahead, a frame looks the same while you scrub, play, present and export.

When nothing moves

• The frame has no unmuted sound of its own. A frame hears its lanes and its videos' sound; a scene in a composition hears the composition's.
• The file is still being analysed. The behaviour's Now bar starts moving when it's done.
• The browser can't decode the file. Convert it to WAV or MP3 and drop it in again.
• Gain is too low or Threshold too high.
• The canvas is at rest. With nothing playing, the signal is 0.

The signals

Every signal runs from 0 to 1 over the file's own range, so a quiet track and a loud one both move the whole way.

The audio behaviour

• Every layer hears the sound at the moment being played, whatever its own time. A layer that starts late, or a motion component on its own clock, still follows the music.
• A frame hears its own lanes and its videos' sound. A scene in a composition hears the composition's sound.
• With nothing playing (the canvas at rest), the signal is 0. A layer shows its own values, so set them to a nice still.
• Gain, From, To and Threshold can be keyed like other behaviour parameters (fade the reaction in, say).
• The agent's edit_behaviours adds and edits them too, with kind: "audio" and a signal.

Visualizers and the shader graph

The visualizers are shader graphs: nodes wired together that PopCraft compiles into an ordinary shader fill.

• Feedback is MilkDrop's trick: echoes of the picture, each zoomed and turned a step further and read a moment earlier. PopCraft works each echo out from the time, not from the frame before. So a frame looks the same however you reach it: scrubbing, looping and exporting all agree.
• A graph's Audio node is its music input. It compiles to a shader param marked @audio, which gets an audio behaviour when the shader is applied. A hand-written shader can declare one too: level: f32, // @range 0 1 @audio bass (or @audio spectrum 0.3 for one band).
• Everything is drawn on the GPU at the fill's size (up to 1024 px on its long side, keeping its proportions).
• The nodes: Coordinate, Time, Value, Colour, Audio; Math, Remap, Smoothstep, Component; Mix, Blend, HSV colour, Colour ramp, Posterize, Brightness; Noise, Gradient, Circle, Plasma, Starfield warp, Rings, Grid; Zoom & rotate, Kaleidoscope, Tunnel, Liquid warp, Mirror, Feedback; Output. Every unwired input is a knob: a keyable shader param named <node>_<input>. See Shader graph.

The Audio reactive library

1. 1.Open Assets and click Use on Audio reactive0:00
2. 2.Drag a component onto the frame0:03
3. 3.Press Space: it follows the music0:05

Description

The Lo-fi loop visualizer, signed in. Assets lists the asset libraries; Use adds Audio reactive, and its motion components show. Spectrum bars is dragged onto the frame, and Space plays the track: the bars dance to the music.

In the asset libraries, Audio reactive has motion components that follow the sound of whatever timeline they are dropped on: Waveform (bars, mirrored or a line of dots), Spectrum bars, Circular spectrum, Pulse ring, VU meter, Beat flash, Bass kick and Audiogram (square and story). Colours and words are inputs.

Music-video templates

New → File → Motion has seven, each with PopCraft's own music (made in code, free to use): Lo-fi loop visualizer, EDM drop teaser, Synthwave tunnel visualizer, Hip-hop lyric video, Indie spectrum poster, Podcast audiogram (1:1 and 9:16) and Festival lineup reveal. Song, artist and lines are text. Colours are brand colours, and the cover art is a photo frame to drop your picture on. Replace the track on the lanes with yours, and every reactive part follows the new sound. Keyed entrances (the lyrics, the lineup) stay where they are. Re-time them to your beats with Detect onsets (Audio).

Presenting

Presenting a frame plays its sound, its audio lanes and its videos' sound, exactly as Animate's Play does. Everything that follows the music follows it there too.

Exporting

MP4 and WebM exports carry the sound, and the picture follows it from the first frame: PopCraft analyses every clip before it draws anything. GIFs and PNG sequences have no sound, but they still show the motion the sound drives.