# Generalists vs. Specialists: Evaluating Large Language Models for Urdu

**Authors:** Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais Athar  
**arXiv:** [2407.04459](https://arxiv.org/abs/2407.04459)  
**Date:** 2024-07-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.04459v3/

## Key points

- Authors conducted a comparative analysis of language models in the context of Urdu
- Study evaluated general-purpose models (GPT-4-Turbo, Llama-3-8b) against specialized models (XLM-Roberta-large, mT5-large)
- Specialized models consistently outperformed general-purpose ones in various tasks related to Urdu
- GPT-4-Turbo's evaluation for generation tasks aligned more closely with human standards compared to Llama-3-8b's evaluations
- Research emphasizes the importance of tailored approaches and optimization of NLP techniques for underrepresented languages like Urdu

## Summary

, , , , 

In their paper titled "Generalists vs. Specialists: Evaluating Large Language Models for Urdu," authors Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, and Awais Athar delve into a comparative analysis of language models in the context of the Urdu language. The study focuses on evaluating general-purpose models such as GPT-4-Turbo and Llama-3-8b against specialized models like XLM-Roberta-large, mT5-large, and Llama-3-8b that have been fine-tuned for specific tasks. With approximately 70 million native speakers, Urdu is an underrepresented language in Natural Language Processing (NLP). The authors conduct a thorough human evaluation for both classification and generation tasks to assess the performance of these models. Their findings reveal that specialized models consistently outperform general-purpose ones in various tasks related to Urdu. Additionally, they note that GPT-4-Turbo's evaluation for generation tasks aligns more closely with human standards compared to Llama-3-8b's evaluations. This research contributes valuable insights to the NLP community by shedding light on the effectiveness of both general and specific-purpose Large Language Models for low-resource languages like Urdu. It highlights the importance of tailored approaches in leveraging language models for specific linguistic contexts and emphasizes the need for further exploration and optimization of NLP techniques in underrepresented languages to enhance inclusivity and accuracy in natural language processing applications.