# Layout-guided Indoor Panorama Inpainting with Plane-aware Normalization

**Authors:** Chao-Chen Gao, Cheng-Hsiu Chen, Jheng-Wei Su, Hung-Kuo Chu  
**arXiv:** [2301.05624](https://arxiv.org/abs/2301.05624)  
**Date:** 2023-01-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.05624v1/

## Key points

- Gao et al. present an end-to-end deep learning framework for indoor panoramic image inpainting.
- Previous inpainting methods have difficulty handling complex structure and texture content in indoor scenes.
- The authors propose to exploit both the global and local context of indoor panorama during the inpainting process.
- They use low-level layout edges as a prior to guide the inpainting model for recovering the global indoor structure.
- A plane-aware normalization module is employed to embed plane-wise style features derived from the layout into the generator, encouraging local texture restoration from adjacent room structures (i.e., ceiling, floor, and walls).
- This method outperforms current state-of-the-art methods on a public panoramic dataset in both qualitative and quantitative evaluations.
- The authors compare their approach with other structure-aware methods such as EdgeConnect and multi-task learning frameworks like LISK, noting that these methods still fail to model long-range structure correspondence such as layout in indoor scenes.
- Limitations of their approach include residuals of shading effect caused by removed furniture that can be hard to segment and model accurately; however, their model still produces more realistic results than PanoDR.
- This work presents a promising approach for improving indoor panorama image inpainting quality by incorporating structure knowledge through exploiting both global and local context during the inpainting process.
- Future work includes adopting a more reasonable segmentation mask of indoor scenes' shading areas to improve results in those shaded scenes.

## Summary

In their paper, "Layout-guided Indoor Panorama Inpainting with Plane-aware Normalization," Gao et al. present an end-to-end deep learning framework for indoor panoramic image inpainting. While previous inpainting methods have shown impressive performance on natural perspective images, they often fail to handle panoramic images, particularly indoor scenes that contain complex structure and texture content. To address this issue, the authors propose to exploit both the global and local context of indoor panorama during the inpainting process. Specifically, the authors take the low-level layout edges estimated from the input panorama as a prior to guide the inpainting model for recovering the global indoor structure. They employ a plane-aware normalization module to embed plane-wise style features derived from the layout into the generator, encouraging local texture restoration from adjacent room structures (i.e., ceiling, floor, and walls). The proposed method outperforms current state-of-the-art methods on a public panoramic dataset in both qualitative and quantitative evaluations. The authors compare their approach with other structure-aware methods such as EdgeConnect and multi-task learning frameworks like LISK. They note that while these methods demonstrate impressive performance in generating realistic results, they still fail to model long-range structure correspondence such as layout in indoor scenes. In addition to presenting experimental results showing superior performance over existing methods on real-world scenes, Gao et al. also discuss limitations of their approach. They note that residuals of shading effect caused by removed furniture can be hard to segment and model accurately; however they show that their model is slightly affected by these residuals but still produces more realistic results than PanoDR. Overall, this work presents a promising approach for improving indoor panorama image inpainting quality by incorporating structure knowledge through exploiting both global and local context during the inpainting process. Future work includes adopting a more reasonable segmentation mask of indoor scenes' shading areas to improve results in those shaded scenes.