# Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction

**Authors:** Jiayang Bai, Zhen He, Shan Yang, Jie Guo, Zhenyu Chen, Yan Zhang, Yanwen Guo  
**arXiv:** [2303.10344](https://arxiv.org/abs/2303.10344)  
**Date:** 2023-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.10344v1/

## Key points

- Predicting indoor lighting from a single perspective image is challenging in computer vision and graphics.
- The problem has been decomposed into three sub-tasks: depth-based image warping, panorama inpainting, and high-dynamic-range (HDR) reconstruction.
- Panorama inpainting is critical for achieving locale-aware and robust prediction.
- Recent methods rely on convolutional neural networks (CNNs), but they struggle to capture long-distance relationships and spatially-varying distortion in spherical signals.
- A local-to-global strategy for large-scale panorama inpainting has been proposed, involving depth-guided local inpainting and a transformer-based network called PanoTransformer.
- This pipeline significantly improves the quality of recovered indoor lighting distribution at any locale and enables high fidelity shading on inserted virtual objects.
- The proposed transformer based network captures distortion-free global features from distorted signals and restores globally consistent structures accordingly.
- A new large scale panorama dataset has also been collected with paired masked input and ground truth images for future research purposes.
- Indoor lighting prediction is challenging
- Problem decomposed into 3 sub-tasks: depth-based image warping, panorama inpainting, HDR reconstruction
- Panorama inpainting critical for locale-aware & robust prediction
- Recent methods rely on CNNs but struggle with long-distance relationships & spatially-varying distortion
- Proposed local-to-global strategy involves depth-guided local inpainting & PanoTransformer network
- Pipeline improves quality of recovered indoor lighting distribution & enables high fidelity shading on virtual objects
- Transformer-based network captures distortion-free global features & restores globally consistent structures
- New large-scale panorama dataset collected with masked input & ground truth images

## Summary

Predicting panoramic indoor lighting from a single perspective image is a challenging problem in computer vision and graphics. To address this issue, researchers have decomposed it into three sub-tasks: depth-based image warping, panorama inpainting, and high-dynamic-range (HDR) reconstruction. Among these sub-tasks, the success of panorama inpainting plays a critical role in achieving locale-aware and robust prediction. Recent methods for panorama inpainting rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, CNNs are limited by their receptive fields and struggle to capture long-distance relationships that are prevalent in panoramas. Additionally, the spatially-varying distortion in spherical signals further increases the difficulty for conventional CNNs. To overcome these issues, a local-to-global strategy for large-scale panorama inpainting has been proposed. This method involves a depth-guided local inpainting applied on the warped panorama to fill small but dense holes caused by pixel stretching during image warping at a given locale. Then, a transformer-based network called PanoTransformer is designed to hallucinate reasonable global structures in the large holes with cubemap projections to avoid distortion. The proposed pipeline significantly improves the quality of recovered indoor lighting distribution at any locale and enables high fidelity and globally coherent shading on inserted virtual objects. Furthermore, it can reproduce fine texture details that are consistent with inserting points on specular surfaces. In summary, this work proposes an effective local-to-global panorama inpainting pipeline that fills missing contents while preserving spatially varying indoor illumination with physically plausible global structures and fine details at any locale. The proposed transformer based network captures distortion free global features from distorted signals and restores globally consistent structures accordingly. A new large scale panorama dataset has also been collected with paired masked input and ground truth images for future research purposes.