Results of the summarizing process for the arXiv paper: 2411.02528v1

In their study "What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length," Tjuatja, Neubig, Linzen, and Hao explore the factors influencing language model (LM) acceptability judgments compared to human judgments. They find that factors such as sequence length and unigram frequency significantly impact LM probabilities, whereas humans are more robust to these effects. The researchers introduce MORCELA, a new linking theory that estimates the optimal level of adjustment for these effects using learned parameters for length and unigram frequency. Their analysis shows that MORCELA outperforms the commonly used SLOR theory in predicting acceptability across two transformer LM families. They also discover that larger models require a lower degree of adjustment for unigram frequency but still need significant adjustments overall. Additionally, they reveal that larger LMs' reduced susceptibility to frequency effects may be attributed to their ability to predict rarer words in context. The authors suggest that evaluations of probability-based LM acceptability judgments should consider model-specific qualities related to factors like frequency and length. By doing so, they believe LMs may be more closely correlated with human judgments than previously assumed. However, there remains a notable gap between the maximum correlation between LMs and human judgments and inter-annotator agreement. Future research could explore additional factors or transformations to enhance the alignment between LMs and human judgment, potentially informing the development of more cognitively plausible models. While their evaluations are limited to English data and annotations by English speakers, the study's findings have broader implications for understanding how different factors influence LM acceptability judgments. The researchers suggest further investigation into integrating these insights into training models that better align with human cognitive processes.

• - Factors such as sequence length and unigram frequency significantly impact LM probabilities
• - Humans are more robust to the effects of sequence length and unigram frequency compared to LMs
• - Introduction of MORCELA, a new linking theory that estimates optimal adjustments for length and unigram frequency
• - MORCELA outperforms the commonly used SLOR theory in predicting acceptability across two transformer LM families
• - Larger models require lower adjustment for unigram frequency but still need significant adjustments overall
• - Larger LMs' reduced susceptibility to frequency effects may be due to their ability to predict rarer words in context
• - Evaluations of probability-based LM acceptability judgments should consider model-specific qualities related to factors like frequency and length
• - There is a gap between the maximum correlation between LMs and human judgments and inter-annotator agreement
• - Future research could explore additional factors or transformations to enhance alignment between LMs and human judgment
• - Study's findings have broader implications for understanding how different factors influence LM acceptability judgments

Summary- The length and frequency of words in a sentence affect how well language models predict the next word. - People are better at understanding sentences with different lengths and word frequencies than language models. - A new theory called MORCELA helps adjust language models to be more accurate with length and word frequency. - MORCELA works better than another theory called SLOR in predicting if a sentence is acceptable. - Bigger language models need less adjustment for common words but still need changes overall. Definitions- Sequence length: The number of words in a sentence or sequence. - Unigram frequency: How often a single word appears in a text or dataset. - Language model (LM): A computer program that predicts the next word in a sequence of text. - Theory: An idea or explanation based on evidence and reasoning.

Introduction: Language models (LMs) have become increasingly popular in natural language processing tasks, such as machine translation and text generation. These models are trained on large amounts of data to predict the probability of a given sequence of words. However, their performance is often evaluated based on acceptability judgments, which measure how well they align with human intuition. In their study "What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length," Tjuatja, Neubig, Linzen, and Hao explore the factors influencing LM acceptability judgments compared to human judgments. Background: The researchers note that previous studies have shown that LMs tend to assign higher probabilities to shorter sequences and more frequent words. This has led to concerns about whether these models truly capture human-like language understanding or simply rely on statistical patterns in the training data. To address this issue, Tjuatja et al. propose a new linking theory called MORCELA (Modeling Orthographic Regularities for Correcting Estimated Language Acceptability), which takes into account both length and frequency effects when predicting acceptability. Methodology: To test their theory, the researchers conducted experiments using two transformer LM families – BERT and GPT-2 – trained on English data. They used three different datasets: one containing sentences from Wikipedia articles, one from movie subtitles, and one from books. The sentences were annotated by English speakers for acceptability using a 5-point scale. Results: The results showed that both length and unigram frequency significantly impact LM probabilities but have little effect on human judgments. This suggests that humans are more robust to these effects than LMs are. Additionally, MORCELA outperformed the commonly used SLOR (Sequence Length-based Offset Rule) theory in predicting acceptability across all datasets. Furthermore, the researchers found that larger LMs require less adjustment for unigram frequency but still need significant adjustments overall compared to smaller LMs. This may be due to the fact that larger models have a better ability to predict rarer words in context, reducing their susceptibility to frequency effects. Implications: The study's findings have important implications for evaluating LMs and understanding their limitations. The researchers suggest that evaluations of probability-based LM acceptability judgments should consider model-specific qualities related to factors like frequency and length. By doing so, they believe LMs may be more closely correlated with human judgments than previously assumed. Moreover, the results highlight the need for further investigation into integrating these insights into training models that better align with human cognitive processes. This could potentially lead to the development of more cognitively plausible models. Limitations: One limitation of this study is its focus on English data and annotations by English speakers. Future research could explore how these findings apply to other languages and cultures. Additionally, while MORCELA outperformed SLOR in predicting acceptability, there is still a notable gap between the maximum correlation between LMs and human judgments and inter-annotator agreement. Further research could investigate additional factors or transformations to enhance this alignment. Conclusion: In conclusion, Tjuatja et al.'s study sheds light on the factors influencing LM acceptability judgments compared to human judgments. They introduce a new linking theory – MORCELA – which takes into account both length and frequency effects when predicting acceptability. Their analysis shows that MORCELA outperforms SLOR in predicting acceptability across two transformer LM families trained on English data. The researchers also reveal that larger LMs require less adjustment for unigram frequency but still need significant adjustments overall due to their ability to predict rarer words in context. Overall, this study highlights the importance of considering model-specific qualities when evaluating LM performance and suggests avenues for future research in developing more cognitively plausible models.