# Continual 3D Convolutional Neural Networks for Real-time Processing of Videos

**Authors:** Lukas Hedegaard, Alexandros Iosifidis  
**arXiv:** [2106.00050](https://arxiv.org/abs/2106.00050)  
**Date:** 2021-05-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.00050v3/

## Key points

- Continual 3D Convolutional Neural Networks (Co3D CNNs) introduced as a novel computational formulation for real-time video processing
- Co3D CNNs operate frame-by-frame, eliminating computational redundancies caused by overlapping clips
- Co3D CNNs effectively reuse preexisting 3D-CNN weights, resulting in reduced floating point operations (FLOPs)
- Experiments conducted on datasets such as Kinetics-400 and Charades show remarkable results:
- CoX3D models achieve state-of-the-art complexity/accuracy trade-offs on Kinetics-400
- FLOP reductions ranging from 12.1x to 15.3x compared to regular X3D models
- Accuracy improvements between 2.3% and 3.8%
- Peak memory consumption reduced by up to 48%
- Transient response of Co3D CNNs during start-up investigated
- Extensive benchmarks performed on publicly available 3D CNNs on hardware

## Summary

The paper introduces Continual 3D Convolutional Neural Networks (Co3D CNNs), a novel computational formulation of spatio-temporal 3D CNNs for real-time processing of videos. Unlike traditional methods that process videos in clips, Co3D CNNs operate frame-by-frame, eliminating the computational redundancies caused by repeated convolutions over frames in overlapping clips. The authors demonstrate that Co3D CNNs can effectively reuse preexisting 3D-CNN weights, resulting in reduced per-prediction floating point operations (FLOPs) proportional to the temporal receptive field while maintaining comparable memory requirements and accuracy. To validate their approach, the researchers conduct experiments using multiple models on datasets such as Kinetics-400 and Charades. The results are remarkable, with CoX3D models achieving state-of-the-art complexity/accuracy trade-offs on Kinetics-400. These models exhibit FLOP reductions ranging from 12.1x to 15.3x compared to regular X3D models, along with accuracy improvements between 2.3% and 3.8%. Additionally, peak memory consumption is reduced by up to 48%. Furthermore, the paper investigates the transient response of Co3D CNNs during start up and performs extensive benchmarks on the processing characteristics of publicly available 3D CNNs on hardware. In summary, this research presents a significant advancement in video processing techniques through the introduction of Continual 3D Convolutional Neural Networks. By processing videos frame-by-frame and reusing preexisting weights, Co3D CNNs achieve impressive reductions in computational requirements while maintaining high accuracy levels. The experimental results demonstrate the superiority of CoX3D models compared to traditional X3D models in terms of complexity/accuracy trade offs on various datasets.