# Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity

**Authors:** Zijiao Chen, Jiaxin Qing, Juan Helen Zhou  
**arXiv:** [2305.11675](https://arxiv.org/abs/2305.11675)  
**Date:** 2023-05-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.11675v1/

## Key points

- Significant progress in reconstructing static images from brain activities using non-invasive techniques
- Reconstructing continuous visual experiences like videos is challenging
- Mind-Video approach utilizes masked brain modeling and multimodal contrastive learning with spatiotemporal attention
- Incorporates co-training with augmented Stable Diffusion model to improve video generation capabilities
- Outperforms previous state-of-the-art methods in semantic accuracy and structural similarity index (SSIM)
- Biologically plausible and interpretable, reflecting established physiological processes
- Attention maps demonstrate reliable decoding of fMRI signals based on biological principles
- Promising approach for reconstructing high-quality videos from fMRI data
- Inter-subject generalization ability needs further exploration
- Potential improvements can be made by utilizing more voxels from the cortex in future iterations of the model.

## Summary

In recent years, significant progress has been made in reconstructing static images from brain activities using non-invasive techniques. However, the reconstruction of continuous visual experiences such as videos remains a challenging task. To address this issue, researchers have proposed a novel approach called Mind-Video that utilizes masked brain modeling and multimodal contrastive learning with spatiotemporal attention to reconstruct high-quality videos from continuous functional magnetic resonance imaging (fMRI) data of the cerebral cortex. The model also incorporates co-training with an augmented Stable Diffusion model to further improve video generation capabilities. Evaluation results show that Mind-Video outperforms previous state-of-the-art methods in terms of semantic accuracy and structural similarity index (SSIM). Furthermore, the model is biologically plausible and interpretable, reflecting established physiological processes. Attention maps generated by the trained model demonstrate reliable decoding of fMRI signals based on biological principles. In conclusion, Mind-Video presents a promising approach for reconstructing high-quality videos from fMRI data by leveraging masked brain modeling and multimodal contrastive learning with spatiotemporal attention. However, its inter-subject generalization ability needs further exploration and potential improvements can be made by utilizing more voxels from the cortex in future iterations of the model.