Test-Time Adaptation via Self-Training with Nearest Neighbor Information

Abstract: Adapting trained classifiers using only online test data is important since it is difficult to access training data or future test data during test time. One of the popular approaches for test-time adaptation is self-training, which fine-tunes the trained classifiers using the classifier predictions of the test data as pseudo labels. However, under the test-time domain shift, self-training methods have a limitation that learning with inaccurate pseudo labels greatly degrades the performance of the adapted classifiers. To overcome this limitation, we propose a novel test-time adaptation method Test-time Adaptation via Self-Training with nearest neighbor information (TAST). Based on the idea that a test data and its nearest neighbors in the embedding space of the trained classifier are more likely to have the same label, we adapt the trained classifier with the following two steps: (1) generate the pseudo label for the test data using its nearest neighbors from a set composed of previous test data, and (2) fine-tune the trained classifier with the pseudo label. Our experiments on two standard benchmarks, i.e., domain generalization and image corruption benchmarks, show that TAST outperforms the current state-of-the-art test-time adaptation methods.