TinyML with ESP32 Tutorial

TinyML brings machine learning (ML) models to microcontrollers, allowing you to embed intelligence in small, low-power devices like the ESP32. This tutorial will guide you through the process of using TinyML with an ESP32, from model training to deployment.

What is TinyML?

TinyML refers to the implementation of ML algorithms on microcontrollers and edge devices with limited resources (low memory, processing power, and power consumption). With TinyML, you can enable devices like the ESP32 to detect patterns, perform sensor analysis, or even recognize speech using neural networks.

Prerequisites

To get started with TinyML on the ESP32, you will need:

• ESP32 Development Board (such as ESP32-WROOM-32 or ESP32-DevKitC)
• Arduino IDE (or PlatformIO)
• TensorFlow Lite for Microcontrollers
• Basic understanding of Python, TensorFlow, and C++
• Sensors (depending on your project, such as accelerometer, microphone, etc.)

Step 1: Setting Up Your Environment

1.1 Install Arduino IDE and ESP32 Board Package

If you haven’t already, download and install the Arduino IDE. Then, follow these steps to install the ESP32 board package:

1. Open Arduino IDE.

2. Go to File > Preferences.

In the Additional Board Manager URLs, add the following link:

https://dl.espressif.com/dl/package_esp32_index.json

3. Go to Tools > Board > Boards Manager.

4. Search for “ESP32” and install the latest package by Espressif Systems.

1.2 Install Necessary Libraries

Install the following Arduino libraries for TensorFlow Lite and ESP32:

1. Open Arduino IDE.
2. Go to Sketch > Include Library > Manage Libraries.
3. Search and install:
   • TensorFlow Lite for Microcontrollers
   • ESP32 Servo (for servo motor control if needed)
   • Adafruit Sensor (for interfacing with sensors)

1.3 Install TensorFlow and TinyML Dependencies

You will need TensorFlow to train your ML models and TensorFlow Lite to convert the model to a format usable by the ESP32.

To install TensorFlow and relevant tools, run:

pip install tensorflow numpy

Additionally, install the TensorFlow Lite converter:

pip install tflite-support

Step 2: Train a Machine Learning Model

For this example, we’ll train a simple model to classify sensor data, such as recognizing different hand gestures using an accelerometer.

2.1 Collect and Preprocess Data

You can use any sensor connected to your ESP32 (such as an accelerometer) to collect training data. Let’s assume you’re collecting accelerometer data to detect gestures.

1. Gather the sensor readings for different gestures (e.g., wave, punch, rotate).
2. Save the data in CSV or JSON format.
3. Preprocess the data by normalizing or scaling it, so that all features (sensor values) are within the same range.

2.2 Train the Model

You can use TensorFlow to train a simple neural network model. Here’s an example of how to build and train a model to classify gestures:

import tensorflow as tf
from tensorflow import keras


# Assume input shape is (3,) representing x, y, z values of accelerometer
model = keras.Sequential([
    keras.layers.InputLayer(input_shape=(3,)),
    keras.layers.Dense(16, activation='relu'),
    keras.layers.Dense(16, activation='relu'),
    keras.layers.Dense(3, activation='softmax')  # 3 output classes for 3 gestures
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train the model with your preprocessed sensor data
model.fit(train_data, train_labels, epochs=10)

# Save the model
model.save("gesture_model.h5")

2.3 Convert the Model to TensorFlow Lite

After training, convert the model to TensorFlow Lite format suitable for microcontrollers:

import tensorflow as tf

# Load the trained model
model = tf.keras.models.load_model("gesture_model.h5")

# Convert the model to TensorFlow Lite format
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the converted model to a .tflite file
with open("gesture_model.tflite", "wb") as f:
    f.write(tflite_model)

Step 3: Deploy the Model on ESP32

3.1 TensorFlow Lite for Microcontrollers Setup

To run the model on the ESP32, you will use TensorFlow Lite for Microcontrollers, which is optimized for running ML models on low-resource devices.

1. Copy the .tflite file to the Arduino project folder.
2. Include the TensorFlow Lite library in your Arduino code.

#include "TensorFlowLite.h"
#include "gesture_model.h"  // Your model header file (convert to C array if needed)

3.2 Writing the ESP32 Code

You can write a simple C++ code to load the model and run inference based on real-time sensor data. Here’s an example for gesture recognition:

#include <TensorFlowLite.h>
#include "gesture_model.h"  // Include the model


// Define input and output dimensions
const int kInputSize = 3;  // x, y, z from accelerometer
const int kOutputSize = 3; // 3 gesture classes

// TensorFlow Lite model setup
tflite::MicroInterpreter* interpreter = nullptr;


void setup() {
  Serial.begin(115200);

  // Load the TensorFlow Lite model
  static tflite::MicroErrorReporter micro_error_reporter;
  const tflite::Model* model = tflite::GetModel(gesture_model_tflite);
 

  static tflite::MicroInterpreter static_interpreter(
      model, resolver, tensor_arena, kTensorArenaSize, &micro_error_reporter);

  interpreter = &static_interpreter;
 
  // Allocate memory
  interpreter->AllocateTensors();
}


void loop() {
  // Get sensor data (e.g., from accelerometer)
  float sensor_data[kInputSize] = {x, y, z};  // Replace with actual sensor readings
  
  // Input the data into the model
  float* input = interpreter->input(0)->data.f;
  for (int i = 0; i < kInputSize; i++) {
    input[i] = sensor_data[i];
  }

  // Run inference
  interpreter->Invoke();
  
  // Get the output (gesture classification)
  float* output = interpreter->output(0)->data.f;
  int predicted_gesture = max_index(output, kOutputSize);

  // Print the detected gesture
  Serial.print("Predicted gesture: ");
  Serial.println(predicted_gesture);
}

// Helper function to get the index of the highest output
int max_index(float* arr, int size) {
  int max_i = 0;
  for (int i = 1; i < size; i++) {
    if (arr[i] > arr[max_i]) {
      max_i = i;
    }
  }
  return max_i;
}

3.3 Upload the Code

1. Connect the ESP32 to your computer.
2. In Arduino IDE, select the correct Board and Port under Tools.
3. Upload the code to your ESP32.

Step 4: Test the Model

After uploading the code, open the Serial Monitor in Arduino IDE. The ESP32 will read the sensor data, run the model, and print the predicted gesture in the Serial Monitor.

Conclusion

You’ve successfully deployed a TinyML model on an ESP32! You can now use this setup for various ML applications like gesture recognition, voice command processing, or anomaly detection.