# Splicing ViT Features for Semantic Appearance Transfer

**Authors:** Narek Tumanyan, Omer Bar-Tal, Shai Bagon, Tali Dekel  
**arXiv:** [2201.00424](https://arxiv.org/abs/2201.00424)  
**Date:** 2022-01-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.00424v1/

## Key points

- Authors present a method for transferring visual appearance between natural images
- Goal is to generate an image where objects in a source image have the appearance of semantically related objects in a target image
- Method uses a generator trained on a single structure/appearance image pair
- Semantic information is incorporated using a pre-trained Vision Transformer (ViT) model as an external semantic prior
- Novel representations of structure and appearance are derived from deep ViT features
- Objective function splices these representations together in the space of ViT features
- Framework called "Splice" does not involve adversarial training or require additional input information such as semantic segmentation or correspondences
- Can generate high-resolution results and works well on various in-the-wild image pairs with variations in objects, pose, and appearance
- DINO-ViT's intermediate representations are explored, showing that the global token (CLS token) encodes texture information and captures global aspects like object parts
- These features provide powerful semantic information at high spatial granularity for reconstructing the original image
- Overall, the method achieves impressive results without complex training procedures or additional input data

## Summary

In their paper titled "Splicing ViT Features for Semantic Appearance Transfer," authors Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel present a method for transferring the visual appearance of one natural image to another. The goal is to generate an image where objects in a source structure image are painted with the visual appearance of their semantically related objects in a target appearance image. The proposed method works by training a generator using only a single structure/appearance image pair as input. To incorporate semantic information into the framework, the authors leverage a pre-trained and fixed Vision Transformer (ViT) model as an external semantic prior. They derive novel representations of structure and appearance from deep ViT features and establish an objective function that splices these representations together in the space of ViT features. Notably, their framework, called "Splice," does not involve adversarial training or require additional input information such as semantic segmentation or correspondences. It can generate high-resolution results and has been demonstrated to produce high-quality outputs on various in-the-wild image pairs, even under significant variations in the number of objects, their pose, and appearance. The authors also explore the intermediate representations learned by DINO-ViT which is known for its powerful visual representation capabilities. Through feature inversion visualization techniques they observe that the global token (CLS token) in ViT encodes not only texture information but also captures more global aspects such as object parts. These features provide powerful semantic information at high spatial granularity and can be used for reconstructing the original image. Overall this paper introduces a novel approach for semantically transferring visual appearances between images using ViT features. The method achieves impressive results without relying on complex training procedures or additional input data making it highly efficient and effective for its purpose.