Project — Spring 2026
An interactive visual and audio-reactive piece that lets users paint on screen through hand gestures, playful movement, and a custom Sonic Pi soundtrack.
Coloring, 2026, 1280px × 720px, TouchDesigner and Sonic Pi
About the Work
Coloring is an interactive TouchDesigner piece that turns hand movement into a digital painting tool. Using a computer camera and MediaPipe hand tracking, the work lets users paint with layered video textures, generated color, and manipulated live camera imagery.
The piece was inspired by the connection between storytelling, gesture, music, and play. Instead of using a mouse or keyboard, users shape the artwork with their hands, making the experience feel physical, expressive, and personal.
The first hand controls the visual painting experience. A pinch between the pointer finger and thumb changes the brush size, a thumbs-up switches between four brush styles, and pointing upward clears the screen for a fresh start. The second hand controls the audio cutoff by pinching, allowing users to shape the custom soundtrack while they paint.
The Process
This patch brings together camera input, MediaPipe tracking, gesture recognition, video manipulation, generated visuals, and audio control. I built the system so each gesture connects directly to a creative action, turning the user's hand into both a brush and a controller.
I coded the soundtrack in Sonic Pi to feel fun, whimsical, and loopable, like background music for a playful video game. The song repeats with slight variations so it supports the interaction without feeling static.
Live Demo & Critique
Coloring was presented during my final class critique, where students tested the patch and created their own playful compositions. The live demo showed how each user brought a different rhythm, gesture, and personality to the piece.