# CogVLM: Visual Expert for Pretrained Language Models

**Authors:** Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang  
**arXiv:** [2311.03079](https://arxiv.org/abs/2311.03079)  
**Date:** 2023-11-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.03079v2/

## Key points

- Authors introduce CogVLM as a powerful open-source visual language foundation model
- CogVLM incorporates a trainable visual expert module in attention and FFN layers to bridge the gap between frozen pretrained language models and image encoders
- Enables deep fusion of vision-language features without compromising performance on natural language processing tasks
- CogVLM-17B has achieved state-of-the-art performance on 10 classic cross-modal benchmarks
- Ranks second on various benchmarks, surpassing or matching PaLI-X 55B
- Codes and checkpoints for CogVLM are available at https://github.com/THUDM/CogVLM

## Summary

In their paper titled "CogVLM: Visual Expert for Pretrained Language Models," authors Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu,
Juanzi Li,Yuxiao Dong,Ming Ding,and Jie Tang introduce CogVLM as a powerful open-source visual language foundation model. This innovative model incorporates a trainable visual expert module in the attention and FFN layers to bridge the gap between frozen pretrained language models and image encoders. Unlike traditional shallow alignment methods used to map image features into the input space of language models,CogVLM enables deep fusion of vision-language features without compromising performance on natural language processing tasks. The authors report that CogVLM-17B has achieved state-of-the-art performance on 10 classic cross-modal benchmarks including NoCaps,Flicker30k captioning ,RefCOCO ,RefCOCO+, RefCOCOg ,Visual7W,GQA ,ScienceQA,VizWiz VQA,and TDIUC.Additionally,it ranks second on VQAv2 OKVQA TextVQA COCO captioning among others,surpassing or matching PaLI-X 55B.The codes and checkpoints for CogVLM are available at https://github.com/THUDM/CogVLM.This groundbreaking approach represents a significant advancement in the field of vision-language fusion and demonstrates the potential for enhancing performance across a range of cross-modal tasks.