Deep contextualized word representations

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   Vector-ICL: In-context Learning with Continuous Vector Representations

   Yufan Zhuang, Chandan Singh, Liyuan Liu et al. · 2024 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL. In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance. In our experiments across various tasks and modalities, including text reconstru…

2. Close Not relevant

   Enriching Word Vectors with Subword Information

   Piotr Bojanowski, Edouard Grave, Armand Joulin et al. · 2016 · Computation and Language · Not summarized yet

   Abstract

   Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character $n$-grams. A vector representation is associated to each character $n$-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word …

3. Close Not relevant

   User-LLM: Efficient LLM Contextualization with User Embeddings

   Lin Ning, Luyang Liu, Jiaxing Wu et al. · 2024 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs) have revolutionized natural language processing. However, effectively incorporating complex and potentially noisy user interaction data remains a challenge. To address this, we propose User-LLM, a novel framework that leverages user embeddings to contextualize LLMs. These embeddings, distilled from diverse user interactions using self-supervised pretraining, capture latent user preferences and their evolution over time. We integrate these user embeddings with LLMs through cross-attention and soft-prompting, enabling LLMs to dynamically adapt to user context. Our comprehensive experiments on MovieLens, Amazon Review, and Google Local Review datasets demonstrate si…

4. Close Not relevant

   LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

   Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach et al. · 2024 · Computation and Language · AI summary available

   Abstract

   Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3 popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed models on English wo…

5. Close Not relevant

   Parallel Context Windows Improve In-Context Learning of Large Language Models

   Nir Ratner, Yoav Levine, Yonatan Belinkov et al. · 2022 · Computation and Language · AI summary available

   Abstract

   For applications that require processing large amounts of text at inference time, Large Language Models (LLMs) are handicapped by their limited context windows, which are typically 2048 tokens. In-context learning, an emergent phenomenon in LLMs in sizes above a certain parameter threshold, constitutes one significant example because it can only leverage training examples that fit into the context window. Existing efforts to address the context window limitation involve training specialized architectures, which tend to be smaller than the sizes in which in-context learning manifests due to the memory footprint of processing long texts. We present Parallel Context Windows (PCW), a method that…

6. Close Not relevant

   Effective Long-Context Scaling of Foundation Models

   Wenhan Xiong, Jingyu Liu, Igor Molybog et al. · 2023 · Computation and Language · AI summary available

   Abstract

   We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5…

7. Close Not relevant

   Multi-sense Definition Modeling using Word Sense Decompositions

   Ruimin Zhu, Thanapon Noraset, Alisa Liu et al. · 2019 · Computation and Language · Not summarized yet

   Abstract

   Word embeddings capture syntactic and semantic information about words. Definition modeling aims to make the semantic content in each embedding explicit, by outputting a natural language definition based on the embedding. However, existing definition models are limited in their ability to generate accurate definitions for different senses of the same word. In this paper, we introduce a new method that enables definition modeling for multiple senses. We show how a Gumble-Softmax approach outperforms baselines at matching sense-specific embeddings to definitions during training. In experiments, our multi-sense definition model improves recall over a state-of-the-art single-sense definition mod…

8. Close Not relevant

   Improving Supervised Bilingual Mapping of Word Embeddings

   Armand Joulin, Piotr Bojanowski, Tomas Mikolov et al. · 2018 · Computation and Language · AI summary available

   Abstract

   Continuous word representations, learned on different languages, can be aligned with remarkable precision. Using a small bilingual lexicon as training data, learning the linear transformation is often formulated as a regression problem using the square loss. The obtained mapping is known to suffer from the hubness problem, when used for retrieval tasks (e.g. for word translation). To address this issue, we propose to use a retrieval criterion instead of the square loss for learning the mapping. We evaluate our method on word translation, showing that our loss function leads to state-of-the-art results, with the biggest improvements observed for distant language pairs such as English-Chinese.…