Wednesday, November 2, 2022
World’s fastest inference engine now supports LSTM-based recurrent neural networks
At Plumerai we enable our customers to perform increasingly complex AI tasks on tiny embedded hardware. We recently observed that more and more of such tasks are using recurrent neural networks (RNNs), in particular RNNs using the long-short-term-memory (LSTM) cell architecture. Example uses of LSTMs are analyzing time-series data coming from sensors like IMUs or microphones, human activity recognition for fitness and health monitoring, detecting if a machine will break down, and speech recognition. This led to us optimizing and extending our support for LSTMs and today we are proud to announce that Plumerai’s deep learning inference software greatly outperforms existing solutions for LSTMs on microcontrollers for all metrics: speed, accuracy, RAM usage, and code size.
To demonstrate this, we selected four common LSTM-based recurrent neural networks and measured latency and memory usage. In the table below, we compare our inference software against the September 2022 version of TensorFlow Lite for Microcontrollers (TFLM for short) with CMSIS-NN enabled. We choose TFLM because it is freely available and widely used. Note that ST’s X-CUBE-AI also supports LSTMs, but that only runs on ST chips and works with 32-bit floating-point, making it much slower.
These are the networks used for testing:
- A simple LSTM model from the TensorFlow Keras RNN guide.
- A weather prediction model that performs time series data forecasting using an LSTM followed by a fully-connected layer.
- A text generation model using a Shakespeare dataset using an LSTM-based RNN with a text-embedding layer and a fully-connected layer.
- A bi-directional LSTM that uses context from both directions of the ’time’ axis, using a total of four individual LSTM layers.
| TFLM latency | Plumerai latency | TFLM RAM | Plumerai RAM | |
|---|---|---|---|---|
| Simple LSTM | 941.4 ms | 189.0 ms (5.0x faster) |
19.3 KiB | 14.3 KiB (1.4x lower) |
| Weather prediction | 27.5 ms | 9.2 ms (3.0x faster) |
4.1 KiB | 1.8 KiB (2.3x lower) |
| Text generation | 7366.0 ms | 1350.5 ms (5.5x faster) |
61.1 KiB | 51.6 KiB (1.2x lower) |
| Bi-directional LSTM | 61.5 ms | 15.1 ms (4.1x faster) |
12.8 KiB | 2.5 KiB (5.1x lower) |
Board: STM32L4R9AI with an Arm Cortex-M4 at 120 MHz with 640 KiB RAM and 2 MiB flash. Similar results were obtained using an Arm Cortex-M7 board.
In the above table we report latency and RAM, which are the most important metrics for most users. The faster you can execute a model, the faster the system can go to sleep, saving power. Microcontrollers are also very memory constrained, making thrifty memory usage crucial. In many cases code size (ROM usage) is also important, and again there we outperform TFLM by a large margin. For example, Plumerai’s implementation of the weather prediction model uses 48 KiB including weights and support code, whereas TFLM uses 120 KiB.
The table above does not report accuracy, because accuracy is not changed by our inference engine. It performs the same computations as TFLM without extra quantization or pruning. Just like TFLM, our inference engine does internal LSTM computations in 16 bits instead of 8 bits to maintain accuracy.
In this blog post we highlight the LSTM feature of Plumerai’s inference software. However, other neural networks are also well supported, very fast, and low on RAM and ROM consumption, without losing accuracy. See our MobileNet blog post or the MLPerf blog post for examples and try out our inference software with your own model.
Besides Arm Cortex-M0/M0+/M4/M7/M33, we also optimize our inference software for Arm Cortex-A, ARC EM, and RISC-V architectures. Get in touch if you want to use the world’s fastest inference software with LSTM support on your embedded device.