Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models

In this study, the researchers propose two innovative strategies to reduce the toxicity of pretrained large language models (LMs) without compromising their utility. The first strategy, called MEDA, involves adding raw toxicity scores as meta-data to the pretraining samples. The second strategy, called INST, adds instructions in natural language form to indicate the toxicity of the samples. To obtain toxicity scores for each sample, the researchers use the widely accepted Commercial PerspectiveAPI. Since LMs have a maximum sequence length that is smaller than the size limit of PerspectiveAPI, they process documents in their dataset into samples of 2000 tokens before obtaining PerspectiveAPI scores. The researchers use a total of 15 datasets for their experiments including Common Crawl, The Pile, Books3, OpenWebText2, Stack Exchange, PubMed Abstracts, Wikipedia Gutenberg (PG-19), BookCorpus2 NIH ExPorter ArXiv GitHub and Pile-CC datasets which contribute to a corpus consisting of 339 billion tokens. The results show that INST strategy performs best in reducing toxicity probability by up to 61% while maintaining accuracy on five benchmark NLP tasks and improving AUC scores on four bias detection tasks by 1.3%. Additionally they demonstrate generalizability of their techniques by scaling number of training samples and model parameters. Overall this study presents an effective approach for controlling toxicity in LMs during pretraining by providing them with explicit instructions and incorporating toxicity scores as meta-data.