Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction

Predicting panoramic indoor lighting from a single perspective image is a challenging problem in computer vision and graphics. To address this issue, researchers have decomposed it into three sub-tasks: depth-based image warping, panorama inpainting, and high-dynamic-range (HDR) reconstruction. Among these sub-tasks, the success of panorama inpainting plays a critical role in achieving locale-aware and robust prediction. Recent methods for panorama inpainting rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, CNNs are limited by their receptive fields and struggle to capture long-distance relationships that are prevalent in panoramas. Additionally, the spatially-varying distortion in spherical signals further increases the difficulty for conventional CNNs. To overcome these issues, a local-to-global strategy for large-scale panorama inpainting has been proposed. This method involves a depth-guided local inpainting applied on the warped panorama to fill small but dense holes caused by pixel stretching during image warping at a given locale. Then, a transformer-based network called PanoTransformer is designed to hallucinate reasonable global structures in the large holes with cubemap projections to avoid distortion. The proposed pipeline significantly improves the quality of recovered indoor lighting distribution at any locale and enables high fidelity and globally coherent shading on inserted virtual objects. Furthermore, it can reproduce fine texture details that are consistent with inserting points on specular surfaces. In summary, this work proposes an effective local-to-global panorama inpainting pipeline that fills missing contents while preserving spatially varying indoor illumination with physically plausible global structures and fine details at any locale. The proposed transformer based network captures distortion free global features from distorted signals and restores globally consistent structures accordingly. A new large scale panorama dataset has also been collected with paired masked input and ground truth images for future research purposes.

Predicting Panoramic Indoor Lighting from a Single Perspective Image

Introduction:

Predicting panoramic indoor lighting from a single perspective image is a challenging problem in computer vision and graphics. To address this issue, researchers have decomposed it into three sub-tasks: depth-based image warping, panorama inpainting, and high-dynamic-range (HDR) reconstruction. Among these sub-tasks, the success of panorama inpainting plays a critical role in achieving locale-aware and robust prediction.

Recent Methods for Panorama Inpainting

Recent Methods for Panorama Inpainting

Convolutional Neural Networks (CNNs):

Recent methods for panorama inpainting rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, CNNs are limited by their receptive fields and struggle to capture long-distance relationships that are prevalent in panoramas. Additionally, the spatially varying distortion in spherical signals further increases the difficulty for conventional CNNs.

Local to Global Strategy

Local to Global Strategy

Depth Guided Local Inpainting: To overcome these issues ,a local -to -global strategy for large -scale panorama inpainting has been proposed . This method involves a depth -guided local inpainting applied on the warped panorama to fill small but dense holes caused by pixel stretching during image warping at a given locale . Then ,a transformer -based network called PanoTransformer is designed to hallucinate reasonable global structuresin the large holes with cubemap projections to avoid distortion . The proposed pipeline significantly improves the quality of recovered indoor lighting distribution at any locale and enables high fidelity and globally coherent shading on inserted virtual objects . Furthermore ,it can reproduce fine texture details that are consistent with inserting points on specular surfaces .

Summary

In Summary : This work proposes an effective local -to -globalpanoramain painting pipeline that fills missing contents while preserving spatially varying indoor illumination with physically plausible global structuresand fine detailsat any locale .The proposed transformer based network captures distortion free global featuresfrom distorted signalsand restores globally consistent structures accordingly .A new large scalepanoramadatasethas also been collectedwith paired masked inputand ground truth imagesfor future research purposes.