# AG3D: Learning to Generate 3D Avatars from 2D Image Collections

**Authors:** Zijian Dong, Xu Chen, Jinlong Yang, Michael J. Black, Otmar Hilliges, Andreas Geiger  
**arXiv:** [2305.02312](https://arxiv.org/abs/2305.02312)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.02312v1/

## Key points

- Zijian Dong and Xu Chen propose a novel method for generating realistic 3D avatars from unstructured 2D image collections.
- The authors' approach is to learn generative models of 3D avatars from abundant unstructured 2D image collections, which can capture shape and deformation of the body and loose clothing.
- The proposed method outperforms previous methods in terms of geometry and appearance, as demonstrated through systematic ablation studies.
- The authors' generator design uses a monolithic approach that models humans holistically in a canonical space using an efficient tri-plane representation.
- The proposed method also incorporates multiple discriminators specialized for improving geometric detail as well as perceptually important regions like the face.
- Normal information is used for guiding geometry in the generative setting by discriminating normal maps rendered from their generative model against off-the-shelf monocular estimators applied to images of human subjects.
- The authors contribute a generative model of articulated 3D humans with state-of-the-art appearance and geometry, a new generator that is efficient and can generate and deform loose clothing, and several specialized discriminators that significantly improve visual and geometric fidelity.

## Summary

In their paper titled "AG3D: Learning to Generate 3D Avatars from 2D Image Collections," Zijian Dong and Xu Chen propose a novel method for generating realistic 3D avatars from unstructured 2D image collections. While there has been significant progress in generative models of human appearance in 2D, creating animated and rendered 3D avatars remains a challenge due to the limited availability and high cost of acquiring 3D training data. The authors' approach is to learn generative models of 3D avatars from abundant unstructured 2D image collections, which can capture shape and deformation of the body and loose clothing. To achieve this goal, the authors propose a new adversarial generative model that captures shape and deformation using a holistic 3D generator and an efficient articulation module. They also integrate multiple discriminators to improve realism by focusing on perceptually important regions such as the face while incorporating geometric cues in the form of predicted 2D normal maps. The proposed method outperforms previous methods in terms of geometry and appearance, as demonstrated through systematic ablation studies. The authors' generator design uses a monolithic approach that models humans holistically in a canonical space using an efficient tri-plane representation. To enable fast volume rendering, they adapt the efficient articulation module Fast-SNARF to their generative setting while further accelerating rendering via empty-space skipping informed by a coarse human body prior. This approach is more flexible than prior methods that base deformations on SMPL, enabling it to model deformations for points far away from the body. The proposed method also incorporates multiple discriminators specialized for improving geometric detail as well as perceptually important regions like the face. Normal information is used for guiding geometry in the generative setting by discriminating normal maps rendered from their generative model against off-the-shelf monocular estimators applied to images of human subjects. The authors demonstrate that this additional normal supervision significantly improves the quality of generated 3D shapes. In summary, the authors contribute a generative model of articulated 3D humans with state-of-the-art appearance and geometry, a new generator that is efficient and can generate and deform loose clothing, and several specialized discriminators that significantly improve visual and geometric fidelity. They plan to release code and models for their method.