# GLM-130B: An Open Bilingual Pre-trained Model

**Authors:** Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, Jie Tang  
**arXiv:** [2210.02414](https://arxiv.org/abs/2210.02414)  
**Date:** 2022-10-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.02414v1/

## Key points

- GLM-130B is a bilingual pre-trained language model with 130 billion parameters.
- It aims to provide an open-source alternative to models like GPT-3 and promote openness and inclusivity in LLM research.
- The reproducibility of GLM-130B is ensured through the release of code, training lessons, and the entire pre-training process.
- GLM-130B can be efficiently run on popular GPUs like RTX 3090 and RTX 2080 Ti, making it accessible to most researchers.
- The paper provides detailed information about the training process, including design choices, training strategies for efficiency and stability, and engineering efforts.
- GLM-130B outperforms GPT-3 175B on various English benchmarks but does not show a performance advantage over OPT-175B and BLOOM-176B.
- It consistently outperforms ERNIE TITAN 3.0 260B, the largest Chinese language model on related benchmarks.
- One unique feature of GLM-130B is its ability to reach INT4 quantization without significant performance loss or specific training methods.
- -The paper emphasizes ethical considerations in LLM research while acknowledging potential risks associated with harmful applications of LLMs.
- -Promoting inclusivity and transparency can help address fairness bias privacy truthfulness issues by providing open access to LLMs.

## Summary

This paper introduces GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. The goal of GLM-130B is to provide an open-source alternative to models like GPT-3 and promote openness and inclusivity in large language model (LLM) research. The reproducibility of GLM-130B is ensured through the release of code, training lessons, and the entire pre-training process. Additionally, GLM-130B can be efficiently run on popular GPUs like RTX 3090 and RTX 2080 Ti, making it accessible to most researchers. The paper provides detailed information about the training process of GLM-130B, including design choices, training strategies for efficiency and stability, and engineering efforts. It highlights the challenges faced during the training process such as loss spikes and disconvergence. GLM-130B outperforms GPT-3 175B on various English benchmarks but does not show a performance advantage over OPT-175B and BLOOM-176B. It also consistently outperforms ERNIE TITAN 3.0 260B, the largest Chinese language model on related benchmarks. One unique feature of GLM-130B is its ability to reach INT4 quantization without quantization aware training or significant performance loss which allows for efficient inference on affordable GPUs like RTX 3090 or RTX 2080 Ti. The paper emphasizes the importance of ethical considerations in LLM research while acknowledging potential risks associated with LLMs being used for harmful applications. They argue that promoting inclusivity and transparency can lead to better defense against such harms by providing open access to LLMs which enables researchers to develop algorithms that can identify synthetic text as well as address fairness bias privacy truthfulness issues.