# Features-based embedding or Feature-grounding

**Authors:** Piotr Makarevich  
**arXiv:** [2506.22442](https://arxiv.org/abs/2506.22442)  
**Date:** 2025-06-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2506.22442v1/

## Key points

- The paper discusses feature-based embedding or feature-grounding in deep learning models
- Everyday reasoning and prior knowledge influence expectations of objects based on their properties
- Approach involves constructing feature-grounded embeddings aligning shareable representations with domain-specific features
- Method includes propagating through the model, computing reconstruction and contrastive losses, and backpropagating through the embedding layer
- Pretraining phase allows extraction and utilization of the embedding layer in downstream tasks
- A pre-training algorithm diagram outlines the process step by step, emphasizing loss computation and embedding layer extraction
- Use case example demonstrates effectiveness by showing alignment in semantic structures across independently trained models with swapped word-embedding layers
- Grounding features are generated using a tokenizer vocabulary derived from BERT, including part-of-speech categories such as nouns, verbs, adjectives, adverbs, etc.
- Incorporating knowledge-based structured thinking into deep learning models through feature-based embeddings can improve overall performance

## Summary

The paper delves into the concept of feature-based embedding or feature-grounding in deep learning models. It focuses on how our everyday reasoning and prior knowledge influence our expectations of objects based on their properties. The study presents a specific approach to constructing feature-grounded embeddings that align shareable representations with interpretable domain-specific conceptual features. The proposed method involves propagating through the model and computing reconstruction and contrastive losses. These losses are then backpropagated through the embedding layer. After a pretraining phase, the embedding layer can be extracted and utilized in downstream tasks. A pre-training algorithm diagram outlines this process step by step, highlighting the computation of losses and extraction of the embedding layer for future use. To demonstrate the effectiveness of this approach, a use case example is provided where models with swapped word-embedding layers trained on different datasets show alignment in semantic structures across independently trained models. In experiments, grounding features are generated using a tokenizer vocabulary derived from BERT. These include part-of-speech categories such as nouns, verbs, adjectives, adverbs, and more. Prior to feature-grounding, binary classification was performed on model pairs to assess performance. The results show that incorporating knowledge-based structured thinking into deep learning models through feature-based embeddings can improve overall performance.