Résultats de recherche pour « Panorama Inpainting »

Predicting panoramic indoor lighting from a single perspective image is a fundamental but highly ill-posed problem in computer vision and graphics. To achieve locale-aware and robust prediction, this problem can be decomposed into three sub-tasks: depth-based image warping, panorama inpainting and high-dynamic-range (HDR) reconstruction, among which the success of panorama inpainting plays a key role. Recent methods mostly rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, they usually achieve suboptimal performance since the missing contents occupy a very large portion in the panoramic space while CNNs are plagued by limited receptive fields. The spatially-varying distortion in the spherical signals further increases the difficulty for conventional CNNs. To address these issues, we propose a local-to-global strategy for large-scale panorama inpainting. In our method, a depth-guided local inpainting is first applied on the warped panorama to fill small but dense holes. Then, a transformer-based network, dubbed PanoTransformer, is designed to hallucinate reasonable global structures in the large holes. To avoid distortion, we further employ cubemap projection in our design of PanoTransformer. The high-quality panorama recovered at any locale helps us to capture spatially-varying indoor illumination with physically-plausible global structures and fine details.

Empty-room panorama inpainting — removing all furniture from an indoor 360° image and reconstructing the bare room — underpins virtual staging and property visualisation, and is almost always trained on synthetic data because real furnished↔empty image pairs barely exist at scale. This research asks a deceptively simple, high-value question: given a fixed synthetic corpus (Structured3D), what actually improves such a model? Holding a paper-faithful pipeline fixed and varying one lever at a time against a fixed eight-room benchmark, I identify the single most effective lever — and show it is data realism.The study cleanly separates what pays from what is already efficient. Data quantity is exhausted quickly — a nineteen-fold increase, from 1,005 to 19,585 exposure-aligned triplets, holds PSNR steady at ~19 dB (19.04 → 19.00 → 19.08) with SSIM, LPIPS and JOD unchanged — so effort spent there is effort that can be saved and reinvested. Geometric conditioning with a ControlNet delivers exactly the structural benefit it should, visibly straightening wall and floor planes. Output resolution is best supplied by super-resolution, which keeps the pipeline efficient on an 8 GB GPU. The decisive, actionable finding is that data content is the lever: augmenting the empty targets with realistic cast shadows raises PSNR from 19.08 to 19.43 dB and improves SSIM, LPIPS and JOD together, across both mask-size regimes. A strong non-generative baseline (LaMa) at 21.76 dB usefully quantifies the headroom that richer, more realistic data stands to unlock.Read together, these controls point to one high-leverage investment: the diversity and realism of the training distribution. The practical consequence, developed in the applications chapter, is encouraging and cost-saving — practitioners of virtual staging, interior design, digital-twin and property-technology systems can achieve the largest gains by investing in data realism (shadow and lighting augmentation, and ultimately real captured pairs), while confidently keeping models, corpora size and resolution lean. The finding is pipeline-agnostic and should transfer to other synthetically-trained indoor inpainting systems.

Single-view scene reconstruction presents significant challenges due to the limited viewpoint information. Panorama images, as a form of image representation with a 360$^\circ$ field of view (FoV), offer a richer source of information for scene reconstruction. Therefore, more and more work utilizes a single panorama image for scene reconstruction. However, existing methods for scene reconstruction from a single panorama image often fail to adequately address the unique characteristics of panorama images and overlook the distinctions between panorama and limited FoV images.In this paper, we propose a novel method for inpainting and reconstructing scenes from a single panorama image. By capitalizing on the distinctive projection characteristics of panorama images and the irregularity of inpainting regions, we develop a panorama inpainting algorithm that incorporates a specialized inpainting strategy and fine-tunes a diffusion model to enhance the quality of inpainting for a single image. Additionally, we exploit the consistency of existing geometry to establish a multi-view consistent denoising process, ensuring coherence among images captured from different viewpoints. Through our two-stage inpainting strategy, we successfully transform the scene geometry derived from a single panorama image into a 3D Gaussian Splatting representation. Comprehensive qualitative and quantitative evaluations demonstrate the superiority of our proposed method compared to state-of-the-art single-view scene reconstruction methods.

The term “attribute transfer” refers to the tasks of altering images in such a way, that the semantic interpretation of a given input image is shifted towards an intended direction, which is quantified by semantic attributes. Prominent example applications are photo realistic changes of facial features and expressions, like changing the hair color, adding a smile, enlarging the nose or altering the entire context of a scene, like transforming a summer landscape into a winter panorama. Recent advances in attribute transfer are mostly based on generative deep neural networks, using various techniques to manipulate images in the latent space of the generator. In this paper, we present a novel method for the common sub-task of local attribute transfers, where only parts of a face have to be altered in order to achieve semantic changes (e.g. removing a mustache). In contrast to previous methods, where such local changes have been implemented by generating new (global) images, we propose to formulate local attribute transfers as an inpainting problem. Removing and regenerating only parts of images, our “Attribute Transfer Inpainting Generative Adversarial Network” (ATI-GAN) is able to utilize local context information to focus on the attributes while keeping the background unmodified resulting in visually sound results.

Neural Radiance Field (NeRF) has achieved substantial progress in novel view synthesis given multi-view images. Recently, some works have attempted to train a NeRF from a single image with 3D priors. They mainly focus on a limited field of view with a few occlusions, which greatly limits their scalability to real-world 360-degree panoramic scenarios with large-size occlusions. In this paper, we present PERF, a 360-degree novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. Notably, PERF allows 3D roaming in a complex scene without expensive and tedious image collection. To achieve this goal, we propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Specifically, we first predict a panoramic depth map as initialization given a single panorama and reconstruct visible 3D regions with volume rendering. Then we introduce a collaborative RGBD inpainting approach into a NeRF for completing RGB images and depth maps from random views, which is derived from an RGB Stable Diffusion model and a monocular depth estimator. Finally, we introduce an inpainting-and-erasing strategy to avoid inconsistent geometry between a newly-sampled view and reference views. The two components are integrated into the learning of NeRFs in a unified optimization framework and achieve promising results. Extensive experiments on Replica and a new dataset PERF-in-the-wild demonstrate the superiority of our PERF over state-of-the-art methods. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.

Image Inpainting is a task that aims to fill in missing regions of corrupted images with plausible contents. Recent inpainting methods have introduced perceptual and style losses as auxiliary losses to guide the learning of inpainting generators. Perceptual and style losses help improve the perceptual quality of inpainted results by supervising deep features of generated regions. However, two challenges have emerged with the usage of auxiliary losses: (i) the time-consuming grid search is required to decide weights for perceptual and style losses to properly perform, and (ii) loss terms with different auxiliary abilities are equally weighted by perceptual and style losses. To meet these two challenges, we propose a novel framework that independently weights auxiliary loss terms and adaptively adjusts their weights within a single training process, without a time-consuming grid search. Specifically, to release the auxiliary potential of perceptual and style losses, we propose two auxiliary losses, Tunable Perceptual Loss (TPL) and Tunable Style Loss (TSL) by using different tunable weights to consider the contributions of different loss terms. TPL and TSL are supersets of perceptual and style losses and release the auxiliary potential of standard perceptual and style losses. We further propose the Auxiliary Weights Adaptation (AWA) algorithm, which efficiently reweights TPL and TSL in a single training process. AWA is based on the principle that the best auxiliary weights would lead to the most improvement in inpainting performance. We conduct experiments on publically available datasets and find that our framework helps current SOTA methods achieve better results.

Semantic scene understanding is an essential task for self-driving vehicles and mobile robots. In our work, we aim to estimate a semantic segmentation map, in which the foreground objects are removed and semantically inpainted with background classes, from a single RGB image. This semantic foreground inpainting task is performed by a single-stage convolutional neural network (CNN) that contains our novel max-pooling as inpainting (MPI) module, which is trained with weak supervision, i.e., it does not require manual background annotations for the foreground regions to be inpainted. Our approach is inherently more efficient than the previous two-stage state-of-the-art method, and outperforms it by a margin of 3% IoU for the inpainted foreground regions on Cityscapes. The performance margin increases to 6% IoU, when tested on the unseen KITTI dataset. The code and the manually annotated datasets for testing are shared with the research community at https://github.com/Chenyang-Lu/semantic-foregroundinpainting.

We develop in this paper a generic Bayesian framework for the joint estimation of motion and recovery of missing data in a damaged video sequence. Using standard maximum a posteriori to variational formulation rationale, we derive generic minimum energy formulations for the estimation of a reconstructed sequence as well as motion recovery. We instantiate these energy formulations and from their Euler-Lagrange Equations, we propose a full multiresolution algorithms in order to compute good local minimizers for our energies and discuss their numerical implementations, focusing on the missing data recovery part, i.e. inpainting. Experimental results for synthetic as well as real sequences are presented. Image sequences and extra material is available at http://image.diku.dk/francois/seqinp.php.