# LLM360: Towards Fully Transparent Open-Source LLMs

**Authors:** Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen  
**arXiv:** [2312.06550](https://arxiv.org/abs/2312.06550)  
**Date:** 2023-12-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.06550v1/

## Key points

- Recent surge in open-source Large Language Models (LLMs) provides options for AI practitioners and researchers
- LLM360 initiative advocates for fully open-sourcing LLMs
- LLM360 makes training code, data, model checkpoints, and intermediate results available to the community
- Goal of LLM360 is to support open and collaborative AI research
- Authors release two 7B parameter LLMs called Amber and CrystalCoder
- Models come with training code, data, intermediate checkpoints, and analyses
- Complete package can be accessed at https://www.llm360.ai
- Authors aim to push boundaries of LLMs by releasing more large-scale models in the future
- Commitment to openness and collaboration enables researchers to build upon existing work without rediscovering details of training process
- LLM360 enhances transparency in AI research while facilitating advancements in language modeling through shared knowledge and resources.

## Summary

The recent surge in open-source Large Language Models (LLMs) has provided AI practitioners and researchers with diverse options for their work. To address the issue of limited transparency into the training process and hinder progress in the field, the authors present LLM360, an initiative that advocates for fully open-sourcing LLMs by making all training code and data, model checkpoints, and intermediate results available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible for everyone. As a first step towards achieving this goal, the authors release two 7B parameter LLMs called Amber and CrystalCoder. These models were pre-trained from scratch and come with their training code, data, intermediate checkpoints, and analyses. The complete package can be accessed at https://www.llm360.ai. By fully open-sourcing these models, the authors aim to push the boundaries of LLMs through an ongoing effort of releasing more large-scale and stronger models in the future. This commitment to openness and collaboration will enable researchers to build upon existing work without having to rediscover details of the training process. Overall, LLM360 strives to enhance transparency in AI research while facilitating advancements in language modeling through shared knowledge and resources.