UE5.0 + TouchDesinger: Real-Time Music Animation

TouchDesigner Real-time audioAnalysis and OSC DAT output | Unreal Engine 5.0 Automatic Animation

TouchDesigner provides audioAnalysis operator, which is very easy to implement and use for any kind of audio input. This operator help me to animate character and sets animation by music.

In this test, there are two parts to get this music animation works. The first one, I used TouchDesigner and the operator 'audioAnalysis' to extract the music data. And the second part is used Unreal Engine 5.0 animation blueprints and blueprints to act the data. In order to connect two programs, i.e. TouchDesigner and Unreal Engine 5.0, I used OSC protocol to communicate between two programs. In TouchDesigner, use DAT and chopexec to send OSC data out. In Unreal Engine, create a OSC server to receive data. Do notice the way OSC server receive data in Unreal Engine 5.0 is different from any earlier Unreal Engine version.

audioAnalysis

audioAnalysis is very easy to use. Just plug in the audio input and it will work. The audio input can be microphone input or audio file.

This operator gives you 9 analysis data, Low, Mid, High, Spectral Centroid, Slow Moving Partials Density, Fast Moving Partials Density, Kick Detection, Snare Detection, Rythm.

For most of my case, it's enough to dynamic music animation.

DAT OSC Output

This DAT is using python as the default programming language. In this example, I only used whileOn function. As you can see in the script whilOn is taking channel, sampleIndex, val and prev and send out via oscout_High1.

AnimBP OSC server

In animation blueprint Event Graph, a OSC server receive OSC message and extract the value float data and store as Hip and Rot Ear value.

AnimGraph get the two values coming from TouchDesigner and plug into Control Rig for this bunny.

In bunny control rig, the key nodes are Exec with Forward Solve so it can receive runtime data. And create two parameters Hip, Rot Ear for animation blueprint. The images below show you the connection of parameters and bones.

Optimization

There is a quite odd to send OSC message DAT speartely to Unreal, as you see the example above 'High' is a only value that be sent out via port 8004. Which is starnge to me, but it seems started with UE5 the OSC receiver only take frame message in this way. I think tha must have a better way to send out all the values with one DAT script. Or maybe it just me took it a wrong approach. If you're reading this article and knowing better way please to tell me.

Last Update: February 12, 2022

Software: Unreal Engine 5.0

OS: Windows 10

Specs: RTX 3080, AMD 3900x, 64GB RAM

Assets:

Bunny - Lilpupinduy
DetectiveOffice - Deep Poly

Reference:

OSC Out DAT | Derivative

Page updated

Google Sites

Report abuse