Search Results for « Vector embeddings »

Vector representations of graphs and relational structures, whether hand-crafted feature vectors or learned representations, enable us to apply standard data analysis and machine learning techniques to the structures. A wide range of methods for generating such embeddings have been studied in the machine learning and knowledge representation literature. However, vector embeddings have received relatively little attention from a theoretical point of view. Starting with a survey of embedding techniques that have been used in practice, in this paper we propose two theoretical approaches that we see as central for understanding the foundations of vector embeddings. We draw connections between the various approaches and suggest directions for future research.

This article introduces embComp, a novel approach for comparing two embeddings that capture the similarity between objects, such as word and document embeddings. We survey scenarios where comparing these embedding spaces is useful. From those scenarios, we derive common tasks, introduce visual analysis methods that support these tasks, and combine them into a comprehensive system. One of embComp's central features are overview visualizations that are based on metrics for measuring differences in the local structure around objects. Summarizing these local metrics over the embeddings provides global overviews of similarities and differences. Detail views allow comparison of the local structure around selected objects and relating this local information to the global views. Integrating and connecting all of these components, embComp supports a range of analysis workflows that help understand similarities and differences between embedding spaces. We assess our approach by applying it in several use cases, including understanding corpora differences via word vector embeddings, and understanding algorithmic differences in generating embeddings.

To face the tight competition in the telecommunication industry, it is important to minimize the rate of customers stopping their service subscription, which is known as customer churn. For that goal, an explainable predictive customer churn model is an essential tool to be owned by a telecommunication provider. In this paper, we developed the explainable model by utilizing the concept of vector embedding in Deep Learning. We show that the model can reveal churning customers that can potentially be converted back to use the previous telecommunication service. The generated vectors are also highly discriminative between the churning and loyal customers, which enable the developed models to be highly predictive for determining whether a customer would cease his/her service subscription or not. The best model in our experiment achieved a predictive performance of 81.16%, measured by the F1 Score. Further analysis on the clusters similarity and t-SNE plot also confirmed that the generated vectors are discriminative for churn prediction.

This work addresses the need for efficient and reusable vector embeddings (VEs) for large-scale RDF Knowledge Graphs, focusing on the widely used DrugBank dataset. The study aims to reduce the computational burden and environmental impact associated with repeatedly generating embeddings for downstream tasks such as link prediction, clustering, and recommendation. To this end, embeddings are generated from the DrugBank Knowledge Graph—comprising over 3.6 million triples, more than 1.5 million entities, and 95 relations—using 25 models implemented in the PyKEEN framework. The dataset is processed into training, validation, and test splits, and embeddings of fixed dimensionality are produced for both entities and relations. The resulting representations are released in multiple formats, including full JSON files, class-partitioned subsets, and an efficient Parquet-based structure, to support scalable querying. Experimental evaluation demonstrates substantial improvements in access time and memory usage when using structured formats, particularly Parquet. Additionally, the study quantifies the carbon footprint of embedding generation, showing that distributing precomputed embeddings can reduce energy consumption by up to 99.37% compared to on-demand recomputation. Overall, the work provides a comprehensive, reusable resource that facilitates research while promoting computational efficiency and environmental sustainability.

Embedding social network data into a low-dimensional vector space has shown promising performance for many real-world applications, such as node classification, node clustering, link prediction and network visualization. However, the information contained in these vector embeddings remains abstract and hard to interpret. Methods for inspecting embeddings usually rely on visualization methods, which do not work on a larger scale and do not give concrete interpretations of vector embeddings in terms of preserved network properties (e.g., centrality or betweenness measures). In this paper, we study and investigate network properties preserved by recent random walk-based embedding procedures like node2vec, DeepWalk or LINE. We propose a method that applies learning to rank in order to relate embeddings to network centralities. We evaluate our approach with extensive experiments on real-world and artificial social networks. Experiments show that each embedding method learns different network properties. In addition, we show that our graph embeddings in combination with neural networks provide a computationally efficient way to approximate the Closeness Centrality measure in social networks.

Retrieval-augmented generation (RAG) is a promising technique that has shown great potential in addressing some of the limitations of large language models (LLMs). One of the major limitations of LLMs is that they can suffer from outdated information, as their training data may not always reflect the latest information. Additionally, LLMs can also be prone to hallucinations, where they generate responses that are not grounded in fact. RAG aims to mitigate these issues by leveraging a database of relevant documents, which are stored as embedding vectors in a high-dimensional space. However, one of the challenges of using high-dimensional embeddings is that they require a significant amount of memory to store. This can be a major issue, especially when dealing with large databases of documents. To alleviate this problem, we propose the use of 4-bit quantization to store the embedding vectors. This involves reducing the precision of the vectors from 32-bit floating-point numbers to 4-bit integers, which can significantly reduce the memory requirements. Our approach has several benefits. Firstly, it significantly reduces the memory storage requirements of the high-dimensional vector database, making it more feasible to deploy RAG systems in resource-constrained environments. Secondly, it speeds up the searching process, as the reduced precision of the vectors allows for faster computation. Our code is available at https://github.com/taeheej/4bit-Quantization-in-Vector-Embedding-for-RAG.

Multi-vector (MV) embeddings have become a powerful paradigm in neural information retrieval (IR), achieving high retrieval accuracy by representing data with multiple vectors and scoring them via the non-linear Chamfer similarity. Despite their widely perceived superiority over single-vector (SV) embeddings which use inner product similarity, to date there is no formal proof that SV similarities cannot approximate MV similarities with the same representation size. Specifically, we ask the following: for any bounded dataset size $n \leq 2^{poly(m)}$, what is the smallest dimension $D$ so that given any collection of MV embeddings $Q_1,\dots,Q_n,X_1,\dots,X_n \subset \mathbb{R}^d$ containing at most $m$ vectors each, there always exist $q_1,\dots,q_n$, $d_1,\dots,d_n \in \mathbb{R}^{D}$ satisfying $|\langle q_i, d_j \rangle - \texttt{Chamfer}(Q_i,X_j)| \leq ε$ for all $i,j$? Recently, the MUVERA algorithm demonstrated that $D = m^{O(1/ε^2)}$ is possible. If improved to $D = md$, this would imply that MV embeddings are no more expressive than SV embeddings. In this paper, we rule out this scenario. Specifically, we prove the existence of a collection of MV embeddings in $\mathbb{R}^d$, each containing at most $m$ vectors, which require single-vector dimension of $D =(ε^2 m)^{Ω(1/ε)}$ to approximate, establishing a strong separation in representation size between MV and SV embeddings. Our proof leverages the Pattern Matrix Method by constructing a hard instance whose Chamfer similarity matrix encodes the $NAND_k$ boolean function. Our results confirm a long-held belief in the IR community: at a fixed representation size, multi-vector embeddings can express similarities which cannot even be approximately represented by single vector embeddings.

Retrieval-Augmented Generation is a technology that enhances large language models by integrating information retrieval. In the industry, inference services based on LLMs are highly sensitive to cost-performance ratio, prompting the need for improving hardware resource utilization in the inference service. Specifically, vector embedding and retrieval processes take up to 20% of the total latency. Therefore, optimizing the utilization of computational resources in vector embeddings is crucial for enhancing the cost-performance ratio of inference processes, which in turn boosts their product competitiveness.In this paper, we analyze the deployment costs of vector embedding technology in inference services, propose a theoretical formula, and determine through the mathematical expression that increasing the capacity to process concurrent queries is the key to reducing the deployment costs of vector embeddings. Therefore, in this paper, we focus on improving the product's capability to process concurrent queries. To optimize concurrency without sacrificing performance, we have designed a queue manager that adeptly offloads CPU peak queries. This manager utilizes a linear regression model to ascertain the optimal queue depths, a critical parameter that significantly influences the efficacy of the system. We further develop a system named WindVE that uses a CPU-NPU heterogeneous architecture to offload peak concurrent queries, which leverages the performance differences between the two processors to effectively manage traffic surges. Through experiments, we compare WindVE to the state-of-the-art vector embedding framework FlagEmbedding, and achieve a concurrency level up to 22.3% higher than the scheme without offloading.

This comprehensive article examines vector embeddings as a fundamental component of modern artificial intelligence systems, detailing their mathematical foundations, key properties, implementation techniques, and practical applications. The article traces the evolution from basic word embeddings to sophisticated transformer-based architectures, highlighting how these representations enable machines to capture and process semantic relationships in human language and visual data. The article encompasses both theoretical frameworks and practical implementations, from the groundbreaking Word2Vec and GloVe models to contemporary developments in multimodal embeddings and dynamic learning systems. The article demonstrates how vector embeddings have revolutionized various domains, including natural language processing, computer vision, and information retrieval, while addressing crucial considerations in computational efficiency and scalability.