GestSynth

In tinyML we have been working with gesture recognition using the onboard IMU of the Arduino Nano 33 BLE Sense. Let me begin by saying that this board has a lot of great feature, sensors, and is low cost to boot however, the board can be a little finicky. From what I understand it has two serial ports one that has access to only the bootloader for flashing the microprocessor and the only has access to the data coming from the onboard sensors. This makes it hard to use with some programs such as MAX/MSP. 

The idea behind this first project was to use three discrete gestures to control some aspect of playback, basically creating a synthesizer that is triggered by the gestures. The three gestures I chose were a flick of the wrist, a vertical circle, and left twitch. Each gesture triggers both a sound and a change in speed that is applied to a background track.

I used Tiny Motion Trainer to capture that data of each gesture. I also used the serial monitor of the Arduino IDE and a Google Colab Notebook to capture the same data, however, I found that the continuous capture method provided by Tiny Motion Trainer provided better data to train the ML model. 

This shows the initial settings for data collection.
I recorded 100 samples for each gesture, this shows the visual out put of some of the individual samples.
This shows the accuracy and the loss of the model after training it on the data.

After training the model I exported the ML model data and the Arduino code.

/**
* TinyML Trainer Example. 
* Version: v005
*
* This is an example generated by Tiny Motion Trainer 
* It is pre-filled with your trained model and your capture settings 
* 
* Required Libraries:
* - TensorFlow Lite for Microcontrollers (tested with v2.4-beta)
* - Arduino_LSM9DS1
*
* Usage:
* - Make sure the Arduino BLE 33 board is installed (tools->board->Board Manager) and connected
* - Make sure you have the required libraries installed
* - Build and upload the sketch
* - Keep the Arduino connected via USB and open the Serial Monitor (Tools->Serial Monitor)
* - The sketch will log out the label with the highest score once detected.
*
* This is meant as an example for you to develop your own code, and is not production ready.
*
**/

//==============================================================================
// Includes
//==============================================================================

#include <Arduino_LSM9DS1.h>
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include <tensorflow/lite/version.h>


//==============================================================================
// Your custom data / settings
// - Editing these is not recommended
//==============================================================================

// This is the model you trained in Tiny Motion Trainer, converted to 
// a C style byte array.
#include "model.h"

// Values from Tiny Motion Trainer
#define MOTION_THRESHOLD 0.2
#define CAPTURE_DELAY 200 // This is now in milliseconds
#define NUM_SAMPLES 100

// Array to map gesture index to a name
const char *GESTURES[] = {
//    "flick", "left", "circle"
"27", "52", "73"
};


//==============================================================================
// Capture variables
//==============================================================================

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

bool isCapturing = false;

// Num samples read from the IMU sensors
// "Full" by default to start in idle
int numSamplesRead = 0;


//==============================================================================
// TensorFlow variables
//==============================================================================

// Global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;

// Auto resolve all the TensorFlow Lite for MicroInterpreters ops, for reduced memory-footprint change this to only 
// include the op's you need.
tflite::AllOpsResolver tflOpsResolver;

// Setup model
const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

// Create a static memory buffer for TensorFlow Lite for MicroInterpreters, the size may need to
// be adjusted based on the model you are using
constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize];


//==============================================================================
// Setup / Loop
//==============================================================================

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
        
  Serial.begin(9600);

  // Wait for serial monitor to connect
  while (!Serial);

  // Initialize IMU sensors
  if (!IMU.begin()) {
    Serial.println("Failed to initialize IMU!");
    while (1);
  }

  // Print out the samples rates of the IMUs
  Serial.print("Accelerometer sample rate: ");
  Serial.print(IMU.accelerationSampleRate());
  Serial.println(" Hz");
  Serial.print("Gyroscope sample rate: ");
  Serial.print(IMU.gyroscopeSampleRate());
  Serial.println(" Hz");

  Serial.println();

  // Get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);
}

void loop() {
  // Variables to hold IMU data
  float aX, aY, aZ, gX, gY, gZ;

  // Wait for motion above the threshold setting
  while (!isCapturing) {
    if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {
     
      IMU.readAcceleration(aX, aY, aZ);
      IMU.readGyroscope(gX, gY, gZ);

      // Sum absolute values
      float average = fabs(aX / 4.0) + fabs(aY / 4.0) + fabs(aZ / 4.0) + fabs(gX / 2000.0) + fabs(gY / 2000.0) + fabs(gZ / 2000.0);
      average /= 6.;

      // Above the threshold?
      if (average >= MOTION_THRESHOLD) {
        isCapturing = true;
        numSamplesRead = 0;
        break;
      }
    }
  }

  while (isCapturing) {

    // Check if both acceleration and gyroscope data is available
    if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {

      // read the acceleration and gyroscope data
      IMU.readAcceleration(aX, aY, aZ);
      IMU.readGyroscope(gX, gY, gZ);

      // Normalize the IMU data between -1 to 1 and store in the model's
      // input tensor. Accelerometer data ranges between -4 and 4,
      // gyroscope data ranges between -2000 and 2000
      tflInputTensor->data.f[numSamplesRead * 6 + 0] = aX / 4.0;
      tflInputTensor->data.f[numSamplesRead * 6 + 1] = aY / 4.0;
      tflInputTensor->data.f[numSamplesRead * 6 + 2] = aZ / 4.0;
      tflInputTensor->data.f[numSamplesRead * 6 + 3] = gX / 2000.0;
      tflInputTensor->data.f[numSamplesRead * 6 + 4] = gY / 2000.0;
      tflInputTensor->data.f[numSamplesRead * 6 + 5] = gZ / 2000.0;

      numSamplesRead++;

      // Do we have the samples we need?
      if (numSamplesRead == NUM_SAMPLES) {
        
        // Stop capturing
        isCapturing = false;
        
        // Run inference
        TfLiteStatus invokeStatus = tflInterpreter->Invoke();
        if (invokeStatus != kTfLiteOk) {
          Serial.println("Error: Invoke failed!");
          while (1);
          return;
        }

        // Loop through the output tensor values from the model
        int maxIndex = 0;
        float maxValue = 0;
        for (int i = 0; i < NUM_GESTURES; i++) {
          float _value = tflOutputTensor->data.f[i];
          if(_value > maxValue){
            maxValue = _value;
            maxIndex = i;
          }
//          Serial.print(GESTURES[i]);
//          Serial.print(": ");
//          Serial.println(tflOutputTensor->data.f[i], 6);
        }
        
//        Serial.print("Winner: ");
        Serial.println(GESTURES[maxIndex]);
        
//        Serial.println();

        // Add delay to not double trigger
        delay(CAPTURE_DELAY);
      }
    }
  }
}

The above code #include "model.h", here is a sample of what that file looks like…it’s 10429 lines long.

const unsigned char model[] = {
  0x1c, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4c, 0x33, 0x00, 0x00, 0x12, 0x00,
  0x1c, 0x00, 0x04, 0x00, 0x08, 0x00, 0x0c, 0x00, 0x10, 0x00, 0x14, 0x00,
  0x00, 0x00, 0x18, 0x00, 0x12, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00,
  0x60, 0xe8, 0x01, 0x00, 0x10, 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00,
  0x2c, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
  0xe4, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xac, 0x00, 0x00, 0x00,
  0x0f, 0x00, 0x00, 0x00, 0x54, 0x4f, 0x43, 0x4f, 0x20, 0x43, 0x6f, 0x6e...}

This code produces a Serial.printLn() resulting in either a “1”, “2”, or “3” output. This out put is then passed into MAX/MSP to control the synthesizer.

A little about the trouble shooting of MAX/MSP, apparently MAX/MSP uses an old protocol that can’t figure out which serial port is transmitting data from the Nano BLE Sense. This basically make it so MAX can not capture the data/output so nothing happens. After hours of research I could find nothing. David Currie and I worked on it for a bit and in the end he had to reach out to Tom Igoe to find a solution. The solution? A simple message attached to the serial object in MAX reading “dtr 1”. This flag tells MAX to look at serial port one on the Nano BLE Sense for the incoming data. 

The final working MAX/MSP patch, note the message of “dtr 1” in the upper left corner.