# Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves

**Authors:** Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, Rio Yokota  
**arXiv:** [2303.01112](https://arxiv.org/abs/2303.01112)  
**Date:** 2023-03-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.01112v1/

## Key points

- Authors present a novel methodology for pre-training vision transformers using circular harmonics
- Superior performance of ExFractalDB-21k compared to ImageNet-21k demonstrates effectiveness of formula-driven supervised learning (FDSL)
- Emphasis on contours over textures in enhancing pre-training process for vision transformers
- Development of VisualAtom-21k dataset through exploration of contour-oriented synthetic datasets design space
- Achieved top-1 accuracy of 83.7% when fine-tuning ViT-Base on ImageNet-1k with VisualAtom-21k
- Potential for continuous improvement in quality over time with synthetic datasets like VisualAtom-21k
- Advantages of FDSL over real images include elimination of privacy concerns, copyright restrictions, labeling costs/errors, and ethical biases
- Study's acceptance at CVPR 2023 highlights significance in advancing pre-training methodologies for vision transformers and potential for further advancements in synthetic dataset quality and performance optimization

## Summary

In their paper titled "Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves," authors Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, and Rio Yokota present a novel methodology for pre-training vision transformers using circular harmonics. The study builds upon the effectiveness of formula-driven supervised learning (FDSL) in pre-training vision transformers. This is demonstrated by the superior performance of ExFractalDB-21k compared to ImageNet-21k. The authors emphasize the importance of contours over textures in enhancing the pre-training process for vision transformers. The research addresses the lack of systematic investigation into why contour-oriented synthetic datasets can achieve comparable accuracy to real datasets. By developing a methodology based on circular harmonics, the authors systematically explore the design space of contour-oriented synthetic datasets. This approach allows for efficient optimization of FDSL parameters and maximization of image variety within the dataset - identified as a critical factor in achieving high accuracy. The resulting dataset, VisualAtom-21k, is utilized for pre-training ViT-Base and achieves a top-1 accuracy of 83.7% when fine-tuning on ImageNet-1k. Remarkably, this level is close to that achieved by JFT-300M pre-training (84.2%), despite VisualAtom-21k containing only 1/14th of the images in JFT-300M. Unlike static datasets like JFT-300M, synthetic datasets like VisualAtom-21k have potential for continuous improvement in quality over time. Furthermore, FDSL offers advantages over real images by eliminating common issues such as privacy concerns, copyright restrictions, labeling costs/errors and ethical biases. The study's acceptance at CVPR 2023 underscores its significance in advancing pre-training methodologies for vision transformers. It also highlights the potential for further advancements in synthetic dataset quality and performance optimization.