# Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models

**Authors:** Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, Geng Xue, Wenxiu Sun, Qiong Yan, Weisi Lin  
**arXiv:** [2311.06783](https://arxiv.org/abs/2311.06783)  
**Date:** 2023-11-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.06783v1/

## Key points

- Multi-modality foundation models, such as GPT-4V, have revolutionized low-level visual perception and understanding tasks
- These models effectively respond to a wide range of natural human instructions
- However, their capabilities are still in the early stages and require further improvement
- A large-scale subjective experiment was conducted to collect real human feedback on low-level vision
- The experiment involved detailed descriptions of the low-level visual appearance of various images, including factors such as clarity, color, and brightness
- The Q-Pathway dataset was created, consisting of 58K detailed human feedbacks on 18,973 images with diverse low-level appearances
- A conversion process using GPT was designed to transform the feedbacks into diverse-format instruction-response pairs, resulting in the Q-Instruct dataset with 200K pairs
- Experimental results have shown that the Q-Instruct dataset consistently enhances low-level perception and understanding abilities across several foundational models
- The datasets can pave the way for future advancements in general intelligence's ability to perceive and understand low level visual appearance and evaluate visual quality like humans do
- Additional context is provided related to composition techniques like the rule of thirds, leading lines, and framing

## Summary

Multi-modality foundation models, such as GPT-4V, have revolutionized low-level visual perception and understanding tasks by effectively responding to a wide range of natural human instructions. However, while these models have shown promising potential in low-level visual tasks, their capabilities are still in the early stages and require further improvement. To enhance these models, we conducted a large-scale subjective experiment that collected a vast number of real human feedback on low-level vision. The experiment involved detailed descriptions of the low-level visual appearance of various images, including factors such as clarity, color, and brightness. Each feedback provided an overall conclusion about the quality of the image and had an average length of 45 words. This resulted in the creation of the Q-Pathway dataset, which consists of 58K detailed human feedbacks on 18,973 images with diverse low-level appearances. To enable foundation models to respond robustly to different types of questions, we designed a conversion process using GPT (Generative Pre-trained Transformer) to transform these feedbacks into diverse-format instruction-response pairs. This resulted in a dataset called Q-Instruct, which includes 200K instruction-response pairs. Experimental results have shown that the Q-Instruct dataset consistently enhances low-level perception and understanding abilities across several foundational models. We believe that our datasets can pave the way for future advancements in general intelligence's ability to perceive and understand low level visual appearance and evaluate visual quality like humans do. For more information about our datasets, model zoo, and demo please visit our website at https://qfuture.github.io/QInstruct/. In addition to this research work, we also provide some additional context related to composition techniques like the rule of thirds, leading lines, and framing. These techniques can further enhance one's understanding of low level visual attributes and improve image quality.