# Flowing from Words to Pixels: A Framework for Cross-Modality Evolution

**Authors:** Qihao Liu, Xi Yin, Alan Yuille, Andrew Brown, Mannat Singh  
**arXiv:** [2412.15213](https://arxiv.org/abs/2412.15213)  
**Date:** 2024-12-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.15213v1/

## Key points

- Authors Qihao Liu, Xi Yin, Alan Yuille, Andrew Brown, and Mannat Singh investigate the impact of diffusion models and flow matching on media generation.
- Traditional approach involves learning a mapping from Gaussian noise to target media distribution.
- Proposed paradigm explores whether flow matching models can directly map one modality's distribution to another without relying on noise distributions or conditioning mechanisms.
- CrossFlow framework utilizes Variational Encoders for input data and enables Classifier-free guidance.
- CrossFlow with a vanilla transformer slightly outperforms standard flow matching for text-to-image tasks.
- CrossFlow demonstrates better scalability with training steps and model size while allowing for meaningful edits in output space through latent arithmetic operations.
- Competitive with or superior to state-of-the-art methods in various cross-modal and intra-modal mapping tasks such as image captioning, depth estimation, and image super-resolution.

## Summary

In their paper titled "Flowing from Words to Pixels: A Framework for Cross-Modality Evolution," authors Qihao Liu, Xi Yin, Alan Yuille, Andrew Brown, and Mannat Singh investigate the impact of diffusion models and flow matching on media generation. The traditional approach involves learning a mapping from a source distribution of Gaussian noise to a target media distribution. However, the authors propose a new paradigm by exploring whether flow matching models can directly map one modality's distribution to another without relying on noise distributions or conditioning mechanisms. This is achieved through their proposed CrossFlow framework, which utilizes Variational Encoders for input data and enables Classifier-free guidance. Surprisingly, experiments show that CrossFlow with a vanilla transformer (without cross attention) slightly outperforms standard flow matching for text-to-image tasks. Additionally, CrossFlow demonstrates better scalability with training steps and model size while allowing for meaningful edits in the output space through latent arithmetic operations. This showcases the versatility of their approach. The authors also demonstrate that CrossFlow is competitive with or superior to state-of-the-art methods in various cross-modal and intra-modal mapping tasks such as image captioning, depth estimation, and image super-resolution. Through their research, Liu et al. aim to accelerate progress in cross-modal media generation and provide a valuable contribution to the field.