# TalkMosaic: Interactive PhotoMosaic with Multi-modal LLM Q&A Interactions

**Authors:** Kevin Li, Fulu Li  
**arXiv:** [2409.13941](https://arxiv.org/abs/2409.13941)  
**Date:** 2024-09-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.13941v2/

## Key points

- Unique approach to raising awareness about environmental protection through the use of images
- Composing images of endangered animals using various car images to highlight impact of car-related pollution on environment and endangered species
- Introduction of interactive photomosaic user interface for easy switching between tile images and original car image
- Development of multimodal custom GPT named TalkMosaic for efficient Q&A interactions with car images
- Optimization of multimodal large language models (LLMs) through sparse attention and quantization techniques for enhanced computational efficiency
- Main contributions: novel user interface, multimodal custom GPT, insights into optimizing causal attention computation in Transformer models
- Practical application demonstrated through prototypes with diverse car images to facilitate environmental awareness and efficient information retrieval

## Summary

, , , , 


In this study, we present a unique approach to raising awareness about environmental protection through the use of images. By composing images of endangered animals using various car images, we aim to highlight the impact of car-related pollution on the environment and endangered species. Our goal is to empower individuals to make informed decisions regarding environmental conservation efforts in relation to automotive practices. To achieve this, we introduce an innovative user interface called interactive photomosaic. This interface allows users to easily switch between tile images in a photomosaic and the original car image with a simple "click and display" operation. Not only does this maximize the information conveyed in a single composed image, but it also automatically saves the original car image for reference. To further enhance user engagement and provide valuable information, we develop a multimodal custom GPT named TalkMosaic by integrating car images and related knowledge into ChatGPT. Users can upload an original car image to TalkMosaic and ask questions about it, such as where to purchase environmentally-friendly tires. This feature enables efficient Q&A interactions with car images. Additionally, we delve into the optimization of multimodal large language models (LLMs) through sparse attention and quantization techniques. We introduce probabilistic FlashAttention and Staircase Adaptive Quantization methods to accelerate inference speed while maintaining model accuracy. Our analysis demonstrates the feasibility and effectiveness of these approaches in enhancing computational efficiency. Our main contributions include presenting a novel user interface for interactive photomosaic, proposing a multimodal custom GPT for inquiry purposes, and providing insights into optimizing causal attention computation in Transformer models. Through implemented prototypes with diverse car images, we showcase the practical application of our approach in facilitating environmental awareness and efficient information retrieval related to cars and environmental protection standards. In summary, our study offers a comprehensive exploration of using AI technologies for environmental advocacy through creative image compositions, interactive interfaces, and optimized computational techniques. By leveraging these advancements, we aim to empower individuals to make informed decisions regarding environmental conservation efforts in relation to automotive practices.