What is a Tonal Paintbrush?

We have designed an paintbrush that allows you to paint sounds in 3D using the Microsoft Kinect and an IOT (Internet of Things) paintbrush. You simply stand in front of the Kinect and click the button on the brush to start recording sounds in 3D space. Furthermore, slide your thumb on the base of the brush to change the pitch or frequency of the sounds created.

How does it work?

There are 2 main components to this project: the desktop component (a computer connected to a Kinect and a set of speakers), and the handheld component (a paintbrush containing a WiFi transceiver, an MCU and a small circuit). The Kinect is used to track the user's hands, and the computer is used to operate the Kinect and produce sounds. Setup: The user holds the paintbrush in their right hand. The paintbrush has a small MCU, a WiFi transceiver and a small circuit containing a button and a potentiometer. The Kinect will face the user, tracking the position of their right hand. When the user presses the button on the paintbrush, the transceiver communicates this to the computer (communication protocol TBA), and the current position of the paintbrush is recorded by the computer. The computer also produces a sound. Now when the paintbrush is placed in the vicinity of the recorded position, the computer plays back the sound (allowing the user to explore their painting).

Authors and Contributors

The creators of the Tonal Paintbrush are:

This project was created as part of Carnegie Mellon's Build 18 festival.