# FoodMem: Near Real-time and Precise Food Video Segmentation

**Authors:** Ahmad AlMughrabi, Adrián Galán, Ricardo Marques, Petia Radeva  
**arXiv:** [2407.12121](https://arxiv.org/abs/2407.12121)  
**Date:** 2024-07-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.12121v1/

## Key points

- Food segmentation in videos is crucial for addressing challenges in health, agriculture, and food biotechnology.
- Existing limitations lead to inaccurate nutritional analysis, inefficient crop management, and suboptimal food processing practices.
- Enhancing segmentation techniques is necessary to improve dietary assessments, agricultural productivity, and overall food production processes.
- The novel framework FoodMem has been developed for high-quality and near-real-time segmentation and tracking of food items in videos using minimal hardware resources.
- FoodMem excels in segmenting food items from 360-degree unbounded scenes with remarkable precision without issues like flickering or prohibitive inference speeds.
- The framework employs a two-phase solution: transformer segmentation phase for initial masks creation and memory-based tracking phase for monitoring food masks in complex scenes.
- Extensive experimentation on challenging scenarios demonstrated superior performance of FoodMem compared to current state-of-the-art models across various conditions and types of foods.
- Benefits of using FoodMem include reduced segmentation noise, elimination of artifacts, completion of missing segments within segmented videos.
- FoodMem achieves a 2.5% increase in mean average precision for food video segmentation while being 58 times faster on average than existing models.
- Authors Ahmad AlMughrabi, Adrián Galán, Ricardo Marques, and Petia Radeva led the development efforts behind FoodMem's creation and validation through rigorous experimentation processes.

## Summary

Food segmentation in videos plays a crucial role in addressing various real-world challenges related to health, agriculture, and food biotechnology. The existing limitations in this field result in inaccurate nutritional analysis, inefficient crop management, and suboptimal food processing practices, ultimately impacting food security and public health. To overcome these challenges, there is a need to enhance segmentation techniques to improve dietary assessments, agricultural productivity, and the overall food production process. In response to these needs, a novel framework called FoodMem has been developed for high-quality and near-real-time segmentation and tracking of food items in videos using minimal hardware resources. This innovative framework is designed to segment food items from video sequences of 360-degree unbounded scenes with remarkable precision. FoodMem addresses the shortcomings of existing semantic segmentation models by consistently generating masks of food portions in video sequences without issues like flickering or prohibitive inference speeds commonly encountered in video processing contexts. The FoodMem framework employs a two-phase solution: a transformer segmentation phase for creating initial segmentation masks and a memory-based tracking phase for monitoring food masks in complex scenes. Through extensive experimentation on challenging scenarios encompassed in new annotated food datasets such as Nutrition5k and Vegetables & Fruits datasets, FoodMem has demonstrated superior performance compared to current state-of-the-art food segmentation models. It excels across various conditions including different camera angles, lighting variations, reflections, scene complexities, and diverse types of foods. The outcomes of utilizing FoodMem include reduced segmentation noise, elimination of artifacts, and completion of missing segments within the segmented videos. The framework enhances the state-of-the-art by achieving a 2.5% increase in mean average precision for food video segmentation while being 58 times faster on average than existing models. With authors Ahmad AlMughrabi, Adrián Galán, Ricardo Marques, and Petia Radeva leading the development efforts behind FoodMem's creation and validation through rigorous experimentation processes outlined in their research study titled "FoodMem: Near Real-time and Precise Food Video Segmentation," this innovative framework represents a significant advancement towards more efficient and accurate food item segmentation in videos for improved applications across various domains related to health, agriculture, and biotechnology.