# Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

**Authors:** Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid  
**arXiv:** [2302.14115](https://arxiv.org/abs/2302.14115)  
**Date:** 2023-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.14115v2/

## Key points

- Vid2Seq: a multi-modal single-stage dense event captioning model
- Pretrained on narrated videos at scale
- Incorporates special time tokens for event boundaries and textual descriptions in the same output sequence
- Uses unlabeled narrated videos for training by reformulating sentence boundaries as pseudo event boundaries and transcribed speech sentences as pseudo captions
- Surpasses state-of-the-art performance on various dense video captioning benchmarks including YouCook2, ViTT, and ActivityNet Captions
- Strong generalization capabilities to tasks such as video paragraph captioning and video clip captioning
- Performs well in few shot settings
- Code available at https://antoyang.github.io/vid2seq.html
- Paper accepted for presentation at CVPR 2023 with 18 pages and six figures

## Summary

In their work titled "Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning," Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic and Cordelia Schmid introduce Vid2Seq - a multi-modal single-stage dense event captioning model. This model is pretrained on narrated videos that are readily available at scale. The Vid2Seq architecture enhances a language model by incorporating special time tokens which enable the prediction of event boundaries and textual descriptions in the same output sequence. To address the limitation of lack of large-scale annotated datasets for training such unified models, the authors leverage unlabeled narrated videos for dense video captioning by reformulating sentence boundaries as pseudo event boundaries and using transcribed speech sentences as pseudo captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset surpasses state-of-the-art performance on various dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. It also demonstrates strong generalization capabilities to tasks such as video paragraph captioning and video clip captioning as well as performing well in few shot settings. The authors provide public access to their code at https://antoyang.github.io/vid2seq.html and their paper has been accepted for presentation at CVPR 2023 consisting of 18 pages with six figures.