Triggering Sounds with Movement Sensors


This tutorial presupposes basic knowledge of object-oriented programming and writing classes in SuperCollider. If you are not familiar with these concepts, you can get a very good introduction here. It also presupposes that you have set up a working environment with MiniBee movement sensors. However, most of the information here can be easily translated to other kinds of movement sensors as well!

Getting the Numbers: The MBData class

At the core of our work with movement sensors is the ability to figure out how much movement energy we generate at any given time, and to use this as an instrument. This information is not immediately available from an accelerometer such as the MiniBee, so we have to massage the numbers a bit. Luckily this is not a very complicated process: it boils down to figuring out (1) where the sensor was a moment ago, (2) figuring out where it is right now, and (3) figuring out the difference between the two. Basic subtraction, in other words.

The MiniBees gives us three datapoints, named respectively x, y and z. When the sensor is attached to the wrist this translates roughly into x = angle of the wrist in relation to the shoulder, y = rotation of the wrist around its own axis, and z = the direct up-and-down movement when holding the wrist level.

Marije Baalman, the designer of the MiniBee sensor system, kindly helped us write the first prototype of two SuperCollider classes that will take care of translating the raw data into something useful, like triggering a sound file or a synthesizer of some sort. The very first thing we need is a method to get data from the MiniBee into the computer, convert it to a more useful range, and then make it available for our various algorithms. The tool for this is the MBData class:

MBData {
    classvar <>resamplingFreq = 20;

    var <>minibeeID;
    var <delta, <x, <y, <z;
    var data, dataMul=15, dataOffset=7.0;
    var prevData;
    var task;
    var oscFunc;

    *new { | minibeeID=10 |
        ^super.newCopyArgs(minibeeID).init;
    }

    init {
        data = 0.0 ! 3;
        prevData = 0.0 ! 3;
        this.createOscFunc;
        this.createTask;
    }

    createOscFunc {
        oscFunc = OSCFunc({|oscdata|
            data = oscdata[2..] * dataMul - dataOffset;
            data = data.clip(0.0, 1.0);
            x = data[0];
            y = data[1];
            z = data[2];
        }, '/minibee/data', argTemplate: [minibeeID]);
    }

    createTask {
        task = TaskProxy.new({
            inf.do {
                this.calcDelta;
                resamplingFreq.reciprocal.wait;
            }
        }).play;
    }

    calcDelta {
        delta = (data - prevData).abs.sum/3;
        prevData = data.copy;
        ^delta;
    }
}

Let's break this down bit by bit: The heart of the algorithm is the createOscFunc method:

createOscFunc {
    oscFunc = OSCFunc({|oscdata|
        data = oscdata[2..] * dataMul - dataOffset;
        data = data.clip(0.0, 1.0);
        x = data[0];
        y = data[1];
        z = data[2];
    }, '/minibee/data', argTemplate: [minibeeID]);
}

The data from the MiniBees are transmitted over the Open Sound Control (OSC) protocol, which is very well supported in SuperCollider. createOscFunc is, as the name suggests, creating an OSC function that responds to an incoming OSC message. In this case it puts the data points from one MiniBee into the data array, which is declared at the top of the class. This is, in turn, scaled with the dataMul and dataOffset variables and restricted to the range 0.0-1.0, before being plugged into the x, y and z variables. These variables can be accessed from outside of the class by using their 'getter' methods (indicated by the < in front of the variable declaration). For example, if I have initialized an MBData object for MiniBee 10:

~mbData = MBData(10);

then the current value of x can be retrieved by doing:

~mbData.x

The next important feature of the MBData class is the calcDelta method, which is a simple way of calculating the amount of total movement energy a MiniBee measures at any given time.

calcDelta {
    delta = (data - prevData).abs.sum/3;
    prevData = data.copy;
    ^delta;
}

The principle at work here is this: data is an array containing the current values ofx, y and z. prevData is an array containing the previous values of x, y and z. From here it is a simple matter of subtracting prevData from data, making sure the thee values are positive by applying the .abs method, .sum the three values and dividing them by 3 to constrain the final result to the 0.0-1.0 range. Finally, we copy the data array to the prevData array, to set it up for the next round of calculations. The end result is a delta variable that we can access like this:

~mbData.delta;

Resampling

The MiniBees are pretty robust, but any number of issues can cause the rate of transmisison to drop: low batteries, long distance between sender and receiver, as well as interference of various kinds can all cause the data flow to slow down. As a result of this you can sometimes get "stuck" delta values. Depending on your code, this can have undesirable results, such as a note hanging indefinitely until the connection is re-established. Because of this, we implement a resampling mechanism to get a steady stream of data that will always reset the delta value to 0.0, no matter what happens on the transmission side of things. This is implemented as a TaskProxy:

createTask {
    task = TaskProxy.new({
        inf.do {
            this.calcDelta;
            resamplingFreq.reciprocal.wait;
        }
    }).play;
}

The resampling frequency is declared in a classvar at the top of the code, and can be changed dynamically according to need.

Putting the Numbers to Work

Now that we have massaged the incoming data stream, we can start using it to make stuff happen! The vehicle for this stuff-making is the MBDeltaTrig class:

MBDeltaTrig {

    classvar <>mbData;
    classvar <>resamplingFreq = 20;

    var <>speedlim, <>threshold;
    var <>minibeeID;
    var <>minAmp, <>maxAmp;
    var <>function;
    var <task;
    var <deltaFunc;

    *new { | speedlim=0.5, threshold=0.1, minibeeID=10, minAmp=0.0, maxAmp=0.3, function |
        ^super.newCopyArgs(speedlim, threshold, minibeeID, minAmp, maxAmp, function).init;
    }

    init {
    }

    createTask {
        task = TaskProxy.new( {
            var free = true;
            inf.do({
                var dt = deltaFunc.delta;
                var xdir = deltaFunc.xdir;
                var ydir = deltaFunc.ydir;
                if(free) {
                    if(dt > threshold){
                        function.value(dt, minAmp, maxAmp);
                        [dt, minibeeID, minAmp, maxAmp].postln;
                        free = false;
                        SystemClock.sched(speedlim,{
                            free = true;
                        });
                    };
                };
                resamplingFreq.reciprocal.wait;
            });
        });
    }

    play { 
        deltaFunc = mbData[minibeeID];
        this.stop;
        this.createTask;
        task.play;
    }

    stop {
        task.stop;
        task = nil;
    }
}

As with our previous class this boils down to a simple idea: (1) get the delta value of a MiniBee, (2) compare it to a threshold value, and (3) trigger a function if delta exceeds the threshold. Also, make sure that a suitable amount of time has passed since the last time the function was triggered, otherwise you might just get 20 triggers per second, which is almost certainly not what you want!

The pulsing heart of the class is the createTask method:

createTask {
    task = TaskProxy.new( {
        var free = true;
        inf.do({
            var dt = deltaFunc.delta;
            var xdir = deltaFunc.xdir;
            var ydir = deltaFunc.ydir;
            if(free) {
                if(dt > threshold){
                    function.value(dt, minAmp, maxAmp);
                    [dt, minibeeID, minAmp, maxAmp].postln;
                    free = false;
                    SystemClock.sched(speedlim,{
                        free = true;
                    });
                };
            };
            resamplingFreq.reciprocal.wait;
        });
    });
}

Again, we are using a TaskProxy to implement the routine. The key here are the lines:

if(dt > threshold){
    function.value(dt, minAmp, maxAmp);
    [dt, minibeeID, minAmp, maxAmp].postln;
    free = false;
    SystemClock.sched(speedlim,{
        free = true;
    });
};

In other words, if the delta value exceeds threshold, trigger function and pass it the values dt, minAmp and maxAmp.

A quick word about the dt value: as you might have noticed in the declaration of this variable, var dt = deltaFunc.delta, we are calling an instance of MBData to get the delta value of our MiniBee. As a consequence, dt becomes the local version of delta. minAmp and maxAmp then proceed to scale the delta value to a suitable amplitude range, and are passed to the class when instantiated. If the function is triggered, set the free variable to false and schedule a SystemClock to reset it to true after a bit of time defined in the speedlim variable. This is the mechanism to lock down the routine so that we don't get hundreds of events every time we exceed the threshold.

The rest of the class are simple household routines to initialize and set variables.

Setting it All Up

A typical setup could look like this:

~mb = (9..16);
~mbData = IdentityDictionary.new;
~mb.do{|id| ~mbData.put(id, MBData.new(id))};
MBDeltaTrig.mbData = ~mbData;

In this case ~mb is an array containing the IDs of 8 MiniBees. ~mbData is an IdentityDictionary containing 8 corresponding MBData objects. Finally we point the MBDeltaTrig classvariable mbData to our newly created MBData objects.

To generate a simple test-tone for MiniBee no. 9 you could then do something like this:

(
 ~sine = {|dt, minAmp, maxAmp| 
    {
        Pan2.ar(
            SinOsc.ar(
                freq: 440, 
                mul: EnvGen.kr(Env.perc(0.1, 2.0))
                ) 
                * dt.linlin(0.0, 1.0, minAmp, maxAmp), 
            0
        )
    }.play
 };

 ~testMB = MBDeltaTrig.new( 
    speedlim: 0.5, 
    threshold: 0.10, 
    minibeeID: 9, 
    minAmp: -32.dbamp, 
    maxAmp: 0.dbamp,
    function: ~sine
    ).play;
)

Happy coding!