Posture Recognition

The MiniBees can be used as a kind of lo-fi posture recognition device by recording their relative positions and then running a comparison routine. This is implemented by a TaskProxy in a very similar manner to the MBDeltaTrig described in the previous tutorial. Don't expect this to produce industry-leading, AI-level facial recognition, though. This is very much the DIY version, but with reasonable thresholds it is still possible to get creative with it!

The requirements for getting this to work is the MBData class and the MBPosRec class. The MBData class is described in the previous tutorial, so no time will be wasted on this here. The MBPosRec class looks like this:

MBPosRec {
    classvar <>data;
    classvar <>resamplingFreq = 20;

    var <>minibeeIDs, <>speedlim, <>threshold, <>action, <recArray, <currentPos, task, free=true;

    *new { | minibeeIDs, speedlim=1, threshold=0.1, action |
        ^super.newCopyArgs(minibeeIDs, speedlim, threshold, action).init;
    }

    init {
        recArray = List.newClear(16);
    }

    getCurrentPos {
        var tmp = nil ! minibeeIDs.size;
        minibeeIDs.do{|id, idx|
            tmp[idx] = [
                data[id].x,
                data[id].y,
            ];
        };
        currentPos = tmp.flat.copy;
    }

    rec {|idx=0|
        this.getCurrentPos;
        recArray.put(idx, currentPos.copy);
    }

    createTask {
        task = TaskProxy{
            inf.do{
                this.getCurrentPos;
                if(free){ this.compare };
                resamplingFreq.reciprocal.wait;
            }
        }
    }

    compare {
        recArray.do{|pos, idx|
            var diff;
            if(pos.notNil){
                diff = (pos - currentPos).abs.sum;
                diff = diff / minibeeIDs.size;
                if (diff < threshold){ 
                    action.value(idx);
                    free = false;
                    SystemClock.sched(speedlim, {
                        free = true;
                    })
                }
            }
        } 
    }

    play {
        this.createTask;
        task.play;
    }

    stop {
        task.stop;
    }

    free {
        task.clear;
    }
}

The most important chunks of code here are the rec and the compare methods.

rec {|idx=0|
    this.getCurrentPos;
    recArray.put(idx, currentPos.copy);
}

compare {
        recArray.do{|pos, idx|
            var diff;
            if(pos.notNil){
                diff = (pos - currentPos).abs.sum;
                diff = diff / minibeeIDs.size;
                if (diff < threshold){ 
                    action.value(idx);
                    free = false;
                    SystemClock.sched(speedlim, {
                        free = true;
                    })
                }
            }
        } 
}

The rec method records the current values of the MiniBees into a 2-dimensional array. This is the posture we are interested in. The position in the outer array becomes the identifier of the posture, so if you for example pass .rec(3) to an MBPosRec object, this will record the posture into the fourth place of the recArray. If you have just initialized this, it will look like this: [nil, nil, nil, [x-value, y-value, z-value], nil, .....], where x, y and z obviously are the actual values measured. Continue recording new positions into the empty slots (or replace existing ones with new ones), and soon enough you will have a bunch of postures ready to be recognized. A note of caution: the cpu burn is not very high, but if you record too many postures you might find it difficult to get consistent results, and you might just get a lot of false positives. This is why recArray is limited to a size of 16. Like I said in the beginning: this is the DIY version of Facebook's facial recognition algorithm.

The compare method proceeds to do the actual work, by iterating through all the sub-arrays contained in recArray, subtracting them from the current position of the Minibees and returning an index if it finds a match. Note that we are not looking for an exact match here! If you would run this algorithm with

if (pos == currentPos)

instead of

if ((pos-currentPos) < threshold)

you would have to get very, very lucky to recognize a posture. Set the threshold too low and you'll hardly ever strike gold, set it too high and you'll trigger all the poses all the time. Experiment to find the right balance!

The way to use this in a program goes something like this:

(
    //make an instrument
    ~scale = [60, 62, 63, 65, 67, 68, 70, 72]; // C minor
    ~sine = {|pose| 
        {
            Pan2.ar(
                SinOsc.ar(
                    freq: ~scale[pose].midicps,
                    mul: EnvGen.kr(Env.perc(0.1, 2.0))
                ) 
                * 0.3,
            0
            )
        }.play
    };
)

// create the posture recognizer
~posrec = MBPosRec(
    minibeeIDs: [9, 10, 11, 12], // an array of 4 MiniBees
    action: ~sine
);
//start it up
~posrec.play;
//strike a pose
~posrec.rec(0);
//and one more...
~posrec.rec(1);
//and one more...
~posrec.rec(2);
//keep going...
~posrec.rec(3);
~posrec.rec(4);
~posrec.rec(5);
~posrec.rec(6);
~posrec.rec(7);

...and that should be more or less that. Now you can play the piano with your arms and legs up in the air. Happy coding!