Detect solfege hand signs using machine learning (live demo)

a project by Ryan Kemmer and Josh Stovall

About

We have developed a system that allows the user to 'play' solfege hand signs as a MIDI instrument. This technology could have practical applications both in music education and creation.

Solfege hand signs are often used in school choirs to teach music theory concepts. Since vocalists cannot actually see the notes on their instrument, hand signs have been developed as a teaching aid.

This system allows students to play the hand signs as an instrument -- recieving auditory feedback. Previously, a student learning solfege would have to sing the note or play another instrument to 'hear' their hand signs.

Libraries

This project uses TensorFlow, MediaPipe, and JZZ.js.

Setup

cd solfege
php -S 0.0.0.0:1234

go to http://localhost:1234

Use

creating data sets

• make a hand sign
• click the button to create training data
• it will wait 1 second, start recording data for 30 seconds, then save a file with the data

Hand Signs

hand sign	name	pitch class (distance from C)	equivalent note
	do	0	C
	di	1	C#
	ra	1	Db
	re	2	D
	ri	3	D#
	me	3	Eb
	mi	4	E
	fa	5	F
	fi	6	F#
	se	6	Gb
	so	7	G
	si	8	G#
	le	8	Ab
	la	9	A
	li	10	A#
	te	10	Bb
	ti	11	B

Contributing

Any contributions you make are greatly appreciated. Any bugs and change requests are to be reported on the issues tab. If you don't like coding, you can contribute by becoming a sponsor.

Questions?

Please write to contact@instrument.bible or visit instrument.bible.