Résultats de recherche pour « Universal Text Embedding Models »

We introduce llama-embed-nemotron-8b, an open-weights text embedding model that achieves state-of-the-art performance on the Multilingual Massive Text Embedding Benchmark (MMTEB) leaderboard as of October 21, 2025. While recent models show strong performance, their training data or methodologies are often not fully disclosed. We aim to address this by developing a fully open-source model, publicly releasing its weights and detailed ablation studies, and planning to share the curated training datasets. Our model demonstrates superior performance across all major embedding tasks -- including retrieval, classification and semantic textual similarity (STS) -- and excels in challenging multilingual scenarios, such as low-resource languages and cross-lingual setups. This state-of-the-art performance is driven by a novel data mix of 16.1 million query-document pairs, split between 7.7 million samples from public datasets and 8.4 million synthetically generated examples from various open-weight LLMs. One of our key contributions is a detailed ablation study analyzing core design choices, including a comparison of contrastive loss implementations, an evaluation of synthetic data generation (SDG) strategies, and the impact of model merging. The llama-embed-nemotron-8b is an instruction-aware model, supporting user-defined instructions to enhance performance for specific use-cases. This combination of top-tier performance, broad applicability, and user-driven flexibility enables it to serve as a universal text embedding solution.

The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for **universal magic words** that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.

Recent advancements in large language models (LLMs) based embedding models have established new state-of-the-art benchmarks for text embedding tasks, particularly in dense vector-based retrieval. However, these models predominantly focus on English, leaving multilingual embedding capabilities largely unexplored. To address this limitation, we present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision. LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks. These components are seamlessly integrated through a minimal set of trainable parameters that act as a connector, effectively transferring the multilingual encoder's language understanding capabilities to the specialized embedding model. Additionally, to comprehensively evaluate multilingual embedding performance, we introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages. Extensive experimental results demonstrate that LUSIFER significantly enhances the multilingual performance across various embedding tasks, particularly for medium and low-resource languages, without requiring explicit multilingual training data. The code and dataset for training are available at: https://github.com/hieum98/lusifer

Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks.It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking.This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation.To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB).MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages.Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date.We find that no particular text embedding method dominates across all tasks.This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks.MTEB comes with open-source code and a public leaderboard at https://github.com/ embeddings-benchmark/mteb.

Text embeddings are crucial for various natural language processing tasks, gaining popularity in both industry and academia. Recent progress in training data and LLMs has greatly enhanced the development of universal text embeddings, aiming for a single unified model that can handle diverse tasks, domains and languages. However, due to biases inherent in popular evaluation benchmarks, certain perspectives or abilities of these models remain unassessed. One such overlooked aspect is the models’ capacity for negation awareness. To address this gap in the existing literature, this paper presents a comprehensive analysis of the negation awareness in state-of-the-art universal text embedding models. Our investigation reveals a substantial deficiency in these models, as they frequently interpret negated text pairs as semantically similar. This flaw undermines their effectiveness in accurately understanding negated statements. To mitigate this issue and enhance the universality of text embeddings, we introduce a lightweight, parameter-free negation adapter. The proposed solution involves a data-efficient and computational-efficient embedding re-weighting method that does not require modifications to the parameters of existing text embedding models. The proposed solution is able to improve text embedding models’ negation awareness significantly on both simple negation understanding task and complex negation understanding task. Furthermore, the proposed solution can also significantly improve the negation awareness of Large Language Model based task-specific high dimensional universal text embeddings. These results not only bridge a key “universality gap” but also pave the way for a modular semantic adapter paradigm toward more universal, robust, and environmentally conscious text embeddings.

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch.We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model.Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets.Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100× more data.We opensource our pretrained models and code 1 .

Multi-modal retrieval becomes increasingly popular in practice.However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information.Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and imageonly data.In this work, we present a new embedding model VISTA for universal multimodal retrieval.Our work brings forth threefold technical contributions.Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings.Secondly, we develop two data generation strategies, which bring highquality composed image-text to facilitate the training of the embedding model.Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data.In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings.Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

Multimodal representation learning models have demonstrated successful operation across complex tasks, and the integration of vision-language models (VLMs) has further enabled embedding models with instruction-following capabilities. However, existing embedding models lack visual-interactive capabilities to specify regions of interest from users (e.g., point, bounding box, mask), which have been explored in generative models to broaden their human-interactive applicability. Equipping embedding models with visual interactions not only would unlock new applications with localized grounding of user intent, which remains unexplored, but also enable the models to learn entity-level information within images to complement their global representations for conventional embedding tasks. In this paper, we propose a novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends the capabilities of the segmentation model and the vision-language model to the realm of representation learning. In VIRTUE, the segmentation model can process visual prompts that pinpoint specific regions within an image, thereby enabling the embedder to handle complex and ambiguous scenarios more precisely. To evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples that aims to retrieve the text caption by jointly considering the entity with a specific object and image scene. VIRTUE consistently achieves a state-of-the-art performance with significant improvements across 36 universal MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.

We present a novel concept of universal text preprocessing and text-embedded programming (PTEP). Preprocessing and text-embedded programming has been widely used in programming languages and frameworks in a fragmented and mutually isolated way. The PTEP ideas can be found in the implementation of the \TeX\ typesetting system; they are prominent in PHP and similar web languages, and finally they are used in the Jupyter data science framework. This paper presents this area of research and related work in a more unified framework, and we describe the implemented system Starfish that satisfies the following novel principles of PTEP: universality, update and replace modes, flexiblity, configurability, and transparency. We describe the operating model and design of Starfish, which is an open-source system implementing universal preprocessing and text-embedded programming in Perl. The system is transparent and its design allows direct implementation in other programming languages as well.