Results of the summarizing process for the arXiv paper: 2303.17878v1

Memory optimization for deep neural network (DNN) inference is crucial in the emerging field of TinyML, which involves deploying DNN inference tasks on tiny, low-power microcontrollers. However, these devices have limited memory and are heavily constrained by the large intermediate run-time buffers required to store activations and other intermediate data during DNN inference. This leads to high memory usage, making it challenging to deploy applications such as audio keyword detection or radar-based gesture recognition on such devices. To address this challenge, a new Fused Depthwise Tiling (FDT) method has been proposed for the memory optimization of DNNs. Compared to existing tiling methods, FDT reduces memory usage without inducing any run time overhead and applies to a larger variety of network layers than existing tiling methods that focus on convolutions. It improves TinyML memory optimization significantly by reducing memory of models where this was not possible before and additionally providing alternative design points for models that show high run time overhead with existing methods. To identify the best tiling configuration for a given model, an end-to-end flow with a new path discovery method has been proposed. This approach applies FDT and existing tiling methods in a fully automated way, including scheduling operations and planning buffer layout in memory. Out of seven evaluated models, FDT achieved significant memory reduction for two models by 76.2% and 18.1% where existing tiling methods could not be applied. Two other models showed significant run time overhead with existing methods, but FDT provided alternative design points with no overhead but reduced memory savings. The study identified five models from a wide range that benefit from fused tiling: Keyword Spotting (KWS), Text Sentiment Analysis (TXT), Magic Wand (MW), PoseNet (POS), and MobileNet V2 SSDLite. All models were quantized to 8 bits, and performance was comparable across different architectures. In conclusion, the proposed FDT method provides an effective solution to optimize memory usage in DNNs for TinyML applications. The end-to-end flow with a new path discovery method enables fully automated tiling configuration and buffer layout planning, making it easier to deploy DNN inference tasks on tiny, low-power microcontrollers.

• - Memory optimization for deep neural network (DNN) inference is crucial in TinyML.
• - TinyML involves deploying DNN inference tasks on tiny, low-power microcontrollers with limited memory.
• - Fused Depthwise Tiling (FDT) method has been proposed for the memory optimization of DNNs.
• - FDT reduces memory usage without inducing any run time overhead and applies to a larger variety of network layers than existing tiling methods that focus on convolutions.
• - An end-to-end flow with a new path discovery method has been proposed to identify the best tiling configuration for a given model.
• - FDT achieved significant memory reduction for two models by 76.2% and 18.1% where existing tiling methods could not be applied.
• - Five models from a wide range benefit from fused tiling: Keyword Spotting (KWS), Text Sentiment Analysis (TXT), Magic Wand (MW), PoseNet (POS), and MobileNet V2 SSDLite.
• - The proposed FDT method provides an effective solution to optimize memory usage in DNNs for TinyML applications.

For a six-year-old kid - People want to make computers that are very small and use very little power. - These computers need to be able to do really complicated things, like recognizing speech or images. - A new way of organizing the information in these tasks has been invented that makes them take up less space in the computer's memory. - This new way doesn't make the computer slower, and it works for lots of different kinds of tasks. - The people who made this new way also figured out how to choose the best way to organize each task so that it takes up as little space as possible. Definitions - Memory optimization: making sure a computer program uses as little memory (space) as possible - Deep neural network (DNN): a type of computer program that can recognize patterns and make decisions based on them - Inference: using a DNN to make predictions or decisions based on input data - TinyML: using DNNs on very small and low-power devices - Microcontroller: a type of small computer used in many electronic devices

Memory Optimization for Deep Neural Network Inference in TinyML Applications

The emerging field of TinyML involves deploying deep neural network (DNN) inference tasks on tiny, low-power microcontrollers. However, these devices have limited memory and are heavily constrained by the large intermediate run-time buffers required to store activations and other intermediate data during DNN inference. This leads to high memory usage, making it challenging to deploy applications such as audio keyword detection or radar-based gesture recognition on such devices. To address this challenge, a new Fused Depthwise Tiling (FDT) method has been proposed for the memory optimization of DNNs.

Background

Memory optimization is crucial for TinyML applications that involve deploying DNN inference tasks on tiny, low-power microcontrollers. These devices have limited memory and are heavily constrained by the large intermediate run-time buffers required to store activations and other intermediate data during DNN inference. This leads to high memory usage, making it challenging to deploy applications such as audio keyword detection or radar-based gesture recognition on such devices.

Fused Depthwise Tiling Method

To address this challenge, a new Fused Depthwise Tiling (FDT) method has been proposed for the memory optimization of DNNs. Compared to existing tiling methods, FDT reduces memory usage without inducing any run time overhead and applies to a larger variety of network layers than existing tiling methods that focus on convolutions. It improves TinyML memory optimization significantly by reducing memory of models where this was not possible before and additionally providing alternative design points for models that show high run time overhead with existing methods. To identify the best tiling configuration for a given model, an end-to-end flow with a new path discovery method has been proposed. This approach applies FDT and existing tiling methods in a fully automated way, including scheduling operations and planning buffer layout in memory. Out of seven evaluated models, FDT achieved significant memory reduction for two models by 76.2% and 18