Python + UE4.27: Hand Tracking Animation - 1

Webcam Video, AI Hand Tracking Model power by Mediapipe, and Real-time Automatic Animation in Unreal Engine 4.27.

The video below is showing the result of using MediaPipe AI model hand-tracking class and transfer via OSC to Unreal Engine 4.27 skeletal mesh for animation. Benefit from AI hand gestures model and libraries. Extracting image data to control animations is the main focus of this test. The original goal is driven more natural looking animation by tracking human performs.

Eagle's wings flopping are controlled by my finger knuckles orientation in real-time. The way it works is followed a pretty simple logic: Image to Data to Animation. And technical methods in this test I used OpenCV. A huge benefit from Google's open source Mediapipe which is a well-trained OpenCV library for public usage. Rather to recompile OpenCV for Unreal Engine, I used python to call and run the functions. And to communicate between these two programs I used OSC protocol to send data. In Unreal Engine, I first created Eagle's animation blueprint as OSC server to receive data. Then created the Eagle's control rig to take control of it's wings. Finally, in the Eagle's animation blueprint plug the OSC data as reference variables into control rig.

Here's the diagram:

STEP 1: OpenCV and Mediapipe

Python code is on my github link below, and this code is not elegance but it works for me at this moment: https://github.com/easylife122/MPhandTrackingOSCout/blob/main/main.py or Expand this section.

Env and Libraries that I used in this test: Python 3.9, OpenCV 2, np, argparse, udp_client

import cv2

import mediapipe as mp

import numpy as np

import argparse

from pythonosc import udp_client

mp_drawing = mp.solutions.drawing_utils

mp_hands = mp.solutions.hands

# Draw Calculate Fingers Angles

joint_list = [[8, 7, 6], [12, 11, 10], [16, 15, 14], [20, 19, 18], [4, 3, 2]]

joint_list[4]

def draw_finger_angles(image, results, joint_list):

# Loop through hands

for hand in results.multi_hand_landmarks:

# Loop through joint sets

for joint in joint_list:

a = np.array([hand.landmark[joint[0]].x, hand.landmark[joint[0]].y]) # First coord

b = np.array([hand.landmark[joint[1]].x, hand.landmark[joint[1]].y]) # Second coord

c = np.array([hand.landmark[joint[2]].x, hand.landmark[joint[2]].y]) # Third coord

radians = np.arctan2(c[1] - b[1], c[0] - b[0]) - np.arctan2(a[1] - b[1], a[0] - b[0])

angle = np.abs(radians * 180.0 / np.pi)

if angle > 180.0:

angle = 360 - angle

cv2.putText(image, str(round(angle, 2)), tuple(np.multiply(b, [640, 480]).astype(int)),

cv2.FONT_HERSHEY_SIMPLEX, 0.5, (15, 15, 15), 1, cv2.LINE_AA)

return image

# Save finger angles data to join_angle

joint_angle = [180,180,180,180,180]

joint_angle[0]

def finger_angles(results, joint_list, joint_angle):

for hand in results.multi_hand_landmarks:

for joint in joint_list:

a = np.array([hand.landmark[joint[0]].x, hand.landmark[joint[0]].y]) # First coord

b = np.array([hand.landmark[joint[1]].x, hand.landmark[joint[1]].y]) # Second coord

c = np.array([hand.landmark[joint[2]].x, hand.landmark[joint[2]].y]) # Third coord

radians = np.arctan2(c[1] - b[1], c[0] - b[0]) - np.arctan2(a[1] - b[1], a[0] - b[0])

angle = np.abs(radians * 180.0 / np.pi)

if angle > 180.0:

angle = 360 - angle

if joint == [8, 7, 6]:

joint_angle[0] = angle

if joint == [12, 11, 10]:

joint_angle[1] = angle

if joint == [16, 15, 14]:

joint_angle[2] = angle

if joint == [20, 19, 18]:

joint_angle[3] = angle

if joint == [4, 3, 2]:

joint_angle[4] = angle

return joint_angle

# Detect Hands and Draw Data

cap = cv2.VideoCapture(1)

with mp_hands.Hands(min_detection_confidence=0.8, min_tracking_confidence=0.5) as hands:

while cap.isOpened():

ret, frame = cap.read()

# BGR 2 RGB

image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

# Flip on horizontal

image = cv2.flip(image, 1)

# Set flag

image.flags.writeable = False

# Detections

results = hands.process(image)

# Set flag to true

image.flags.writeable = True

# RGB 2 BGR

image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)

# Detections

print(results)

# Rendering results

if results.multi_hand_landmarks:

for num, hand in enumerate(results.multi_hand_landmarks):

mp_drawing.draw_landmarks(image, hand, mp_hands.HAND_CONNECTIONS,

mp_drawing.DrawingSpec(color=(50, 50, 50), thickness=2, circle_radius=4),

mp_drawing.DrawingSpec(color=(250, 50, 50), thickness=2, circle_radius=2),

)

# Draw angles to image from joint list

draw_finger_angles(image, results, joint_list)

finger_angles(results, joint_list, joint_angle)

print(joint_angle)

# Save our image

# cv2.imwrite(os.path.join('Output Images', '{}.jpg'.format(uuid.uuid1())), image)

cv2.imshow('Hand Tracking', image)

# Send it via OSC

if __name__ == "__main__":

parser = argparse.ArgumentParser()

parser.add_argument("--ip", default="127.0.0.1", help="The ip of the OSC server")

parser.add_argument("--port", type=int, default=5005, help="The port the OSC server is listening on")

args = parser.parse_args()

# Send joint_angle

client = udp_client.SimpleUDPClient(args.ip, args.port)

client.send_message("/handOSC", joint_angle)

# for x in range(10):

# client.send_message("/o", joint_angle)

# time.sleep(0)

if cv2.waitKey(10) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

STEP 2: Eagle Animation Blueprint (OSC Receiver & Animation Control)

Here's Event Graph of Animation Blueprint. Running function as OSC server and animation controller. For Control Rig side, will need to create few reference variables and make them editable. In the graph, there are two variables running for transporting purpose: Index and Thumb. They are controlling the wings flopping angles.

Here's the animation graph. Place the control rig between Input Pose and Output Pose then plug in that two parameters Index and Thumb to the Eagle Control Rig. (Need to check the control rig pins of the WRotation_L and WRotation_R in details panel)

STEP 3: Eagle Control Rig

Here I only show the Forward Solve the main function part. I create two reference float parameters: Get WRotation_R and Get WRotation_L. And also create two controller first setup at the shoulders joint position and orientation: WingR and WingL. Plug in GetWRotation_R to control Rotation Y of WingR to control r_shoulder and the same as Left side.

Last Update: February 4, 2022

Software: Unreal Engine 4.27, Python 3.9, Visual Studio Code

OS: Windows 10

Specs: RTX 3080, AMD 3900x, 64GB RAM

Reference:

Unreal Engine OSC plugin - https://docs.unrealengine.com/4.27/en-US/WorkingWithAudio/OSC/
Mediapipe - https://google.github.io/mediapipe/
Python-osc - https://pypi.org/project/python-osc/
Nicholas Renotte's youtube video about Hand Tracking - https://youtu.be/vQZ4IvB07ec