# CogVLM: Visual Expert for Pretrained Language Models

**Authors:** Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang  
**arXiv:** [2311.03079](https://arxiv.org/abs/2311.03079)  
**Date:** 2023-11-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.03079v1/

## Key points

- Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu , Juanzi Li , Yuxiao Dong , Ming Ding and Jie Tang have introduced CogVLM - a powerful open-source visual language foundation model.
- CogVLM incorporates a trainable visual expert module in the attention and FFN layers to bridge the gap between image features and language models.
- CogVLM-17B has achieved state-of-the-art performance on 10 classic cross-modal benchmarks including NoCaps , Flicker30k captioning , RefCOCO , RefCOCO+ , RefCOCOg , Visual7W , GQA ScienceQA , VizWiz VQA and TDIUC .
- It ranks second on VQAv2 , OKVQA , TextVQA , COCO captioning etc., surpassing or matching PaLI-X 55B.
- The codes and checkpoints for CogVLM are available at https://github.com/THUDM/CogVLM.

## Summary

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu , Juanzi Li , Yuxiao Dong , Ming Ding and Jie Tang have introduced CogVLM - a powerful open-source visual language foundation model. Unlike the popular shallow alignment method that maps image features into the input space of a language model without considering their interactions with each other; CogVLM bridges this gap by incorporating a trainable visual expert module in the attention and FFN layers. This enables deep fusion of vision language features without compromising performance on NLP tasks. CogVLM-17B has achieved state-of-the-art performance on 10 classic cross-modal benchmarks including NoCaps , Flicker30k captioning , RefCOCO , RefCOCO+ , RefCOCOg , Visual7W , GQA ScienceQA , VizWiz VQA and TDIUC . It also ranks second on VQAv2 , OKVQA , TextVQA , COCO captioning etc., surpassing or matching PaLI-X 55B. The codes and checkpoints for CogVLM are available at https://github.com/THUDM/CogVLM.