SketchyCOCO: Image Generation from Freehand Scene Sketches

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cited by this paper Used in depth by this paper A method it uses Not relevant

   SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

   Wengling Chen, James Hays · 2018 · Computer Vision and Pattern Recognition · cited 346 times · AI summary available

   Abstract

   Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision. Existing approaches either need exact edge maps, or require a database to retrieve images from. In this work, we propose a novel Generative Adversarial Network (GAN) approach that synthesizes realistic looking images from 50 categories including motorcycles, horses and couches. We demonstrate a data augmentation technique for sketches which is fully automatic, and we show that the augmented data is helpful to our task. We introduce a new building block suit for both the generator and discriminator which improves the information flow and utilizes input images at multiple scales. C…

2. Very close Cites this paper Uses this paper in depth Not relevant

   FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context

   P. Chowdhury, Aneeshan Sain, Yulia Gryaditskaya et al. · 2022 · European Conference on Computer Vision · cited 84 times · Not summarized yet

   Abstract

   We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. With practical applications in mind, we collect sketches that convey scene content well but can be sketched within a few minutes by a person with any sketching skills. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offering both object- and scene-level abstraction. Each sketch is augmented with its text description. Using our dataset, we study for the first time the problem of fine-grained image retrieval from freehand scene sketches and sketch captions. We draw insights on: (i) Scene salience encoded in sketches us…

3. Very close Cites this paper Not relevant

   Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

   Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain et al. · 2023 · Computer Vision and Pattern Recognition · cited 74 times · AI summary available

   Abstract

   Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch. Our contribution at the outset is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only. This importantly ensures that generated results are always photorealistic. The rest…

4. Very close Not relevant

   Face Sketch to Image Generation Using Pix2Pix GAN

   Arthik Shetty, Sachidanand C G, M. Srividya · 2025 · 2025 9th International Conference on Computational System and Information Technology for Sustainable Solutions (CSITSS) · cited 1 time · Not summarized yet

   Abstract

   This work presents an automated image synthesis method from hand-drawn sketches using adversarial networks. Conventional translation approaches enforce rigid constraints forcing outputs to match input sketch edges exactly, producing unrealistic results when sketches are imperfect. Our technique utilizes sketches as soft guidance instead of hard constraints, generating more natural images. We introduce a joint completion framework treating sketches as contextual cues for image synthesis. A contextual GAN learns unified sketch-image distributions through shared representation. Key advantages include: (1) joint representation simplifies learning in common sketch-image space, avoiding cross-doma…

5. Very close Cites this paper Not relevant

   SketchingReality: From Freehand Scene Sketches To Photorealistic Images

   Ahmed Bourouis, Mikhail Bessmeltsev, Yulia Gryaditskaya · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Recent years have witnessed remarkable progress in generative AI, with natural language emerging as the most common conditioning input. As underlying models grow more powerful, researchers are exploring increasingly diverse conditioning signals, such as depth maps, edge maps, camera parameters, and reference images, to give users finer control over generation. Among different modalities, sketches are a natural and long-standing form of human communication, enabling rapid expression of visual concepts. Previous literature has largely focused on edge maps, often misnamed'sketches', yet algorithms that effectively handle true freehand sketches, with their inherent abstraction and distortions, r…

6. Very close Not relevant

   Staged Sketch-to-Image Synthesis via Semi-supervised Generative Adversarial Networks

   Zeyu Li, Cheng Deng, Er-Kun Yang et al. · 2021 · IEEE transactions on multimedia · cited 45 times · Not summarized yet

   Abstract

   Sketch-based image synthesis is a challenging problem in computer graphics and vision. Existing approaches either require exact edge maps or rely on the retrieval of existing photographs, which limits their applications in real-world scenarios. Accordingly in this work, we propose a staged semi-supervised generative adversarial networks based method for sketch-to-image synthesis, which can directly generate realistic images from novice sketches. More specifically, we first adopt a conditional generative adversarial network (CGAN) to extract class-wise representations from unpaired images. These class-wise representations are then exploited and incorporated with another CGAN, which are used t…

7. Very close Not relevant

   Content-Conditioned Generation of Stylized Free-Hand Sketches

   Jiajun Liu, Si-Yuan Wang, Guang-Ming Zhu et al. · 2023 · 2023 International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD) · Not summarized yet

   Abstract

   In recent years, the recognition of free-hand sketches has remained a popular task. However, in some special fields such as the military field, free-hand sketches are difficult to sample on a large scale. Common data augmentation and image generation techniques are difficult to produce images with various free-hand sketching styles. Therefore, the recognition and segmentation tasks in related fields are limited. In this paper, we propose a novel adversarial generative network that can accurately generate realistic free-hand sketches with various styles. We explore the performance of the model, including using styles randomly sampled from a prior normal distribution to generate images with va…

8. Very close Cites this paper Not relevant

   Unsupervised Scene Sketch to Photo Synthesis

   Jiayun Wang, Sangryul Jeon, Stella X. Yu et al. · 2022 · ECCV Workshops · cited 5 times · Not summarized yet

   Abstract

   Sketches make an intuitive and powerful visual expression as they are fast executed freehand drawings. We present a method for synthesizing realistic photos from scene sketches. Without the need for sketch and photo pairs, our framework directly learns from readily available large-scale photo datasets in an unsupervised manner. To this end, we introduce a standardization module that provides pseudo sketch-photo pairs during training by converting photos and sketches to a standardized domain, i.e. the edge map. The reduced domain gap between sketch and photo also allows us to disentangle them into two components: holistic scene structures and low-level visual styles such as color and texture.…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Image Generation from Sketch Constraint Using Contextual GAN

   Yong-Yi Lu, Shangzhe Wu, Yu-Wing Tai et al. · 2017 · European Conference on Computer Vision · cited 149 times · Not summarized yet

   Abstract

   In this paper we investigate image generation guided by hand sketch. When the input sketch is badly drawn, the output of common image-to-image translation follows the input edges due to the hard condition imposed by the translation process. Instead, we propose to use sketch as weak constraint, where the output edges do not necessarily follow the input edges. We address this problem using a novel joint image completion approach, where the sketch provides the image context for completing, or generating the output image. We train a generated adversarial network, i.e, contextual GAN to learn the joint distribution of sketch and the corresponding image by using joint images. Our contextual GAN ha…

2. Close Used in depth by this paper A method it uses Not relevant

   Image Generation from Scene Graphs

   Justin Johnson, Agrim Gupta, Li Fei-Fei · 2018 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · cited 942 times · Not summarized yet

   Abstract

   To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation…

3. Close Used in depth by this paper A method it uses Not relevant

   StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

   Han Zhang, Tao Xu, Hongsheng Li et al. · 2016 · IEEE International Conference on Computer Vision · cited 2,996 times · Not summarized yet

   Abstract

   Synthesizing high-quality images from text descriptions is a challenging problem in computer vision and has many practical applications. Samples generated by existing textto- image approaches can roughly reflect the meaning of the given descriptions, but they fail to contain necessary details and vivid object parts. In this paper, we propose Stacked Generative Adversarial Networks (StackGAN) to generate 256.256 photo-realistic images conditioned on text descriptions. We decompose the hard problem into more manageable sub-problems through a sketch-refinement process. The Stage-I GAN sketches the primitive shape and colors of the object based on the given text description, yielding Stage-I low…

4. Close Used in depth by this paper A method it uses Not relevant

   Specifying Object Attributes and Relations in Interactive Scene Generation

   O. Ashual, Lior Wolf · 2019 · IEEE International Conference on Computer Vision · cited 194 times · Not summarized yet

   Abstract

   We introduce a method for the generation of images from an input scene graph. The method separates between a layout embedding and an appearance embedding. The dual embedding leads to generated images that better match the scene graph, have higher visual quality, and support more complex scene graphs. In addition, the embedding scheme supports multiple and diverse output images per scene graph, which can be further controlled by the user. We demonstrate two modes of per-object control: (i) importing elements from other images, and (ii) navigation in the object space by selecting an appearance archetype. Our code is publicly available at https://www.github.com/ashual/scene_generation.

5. Close A method it uses Not relevant

   A Neural Representation of Sketch Drawings

   David R Ha, Douglas Eck · 2017 · International Conference on Learning Representations · cited 1,031 times · Not summarized yet

   Abstract

   We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Back To The Drawing Board: Rethinking Scene-Level Sketch-Based Image Retrieval

   Emil Demi'c, L. Zajc · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Abstract

   The goal of Scene-level Sketch-Based Image Retrieval is to retrieve natural images matching the overall semantics and spatial layout of a free-hand sketch. Unlike prior work focused on architectural augmentations of retrieval models, we emphasize the inherent ambiguity and noise present in real-world sketches. This insight motivates a training objective that is explicitly designed to be robust to sketch variability. We show that with an appropriate combination of pre-training, encoder architecture, and loss formulation, it is possible to achieve state-of-the-art performance without the introduction of additional complexity. Extensive experiments on a challenging FS-COCO and widely-used Sketc…

2. Close Uses this paper in depth Not relevant

   CoGS: Controllable Generation and Search from Sketch and Style

   Cusuh Ham, Gemma Canet Tarrés, Tu Bui et al. · 2022 · European Conference on Computer Vision · cited 24 times · Not summarized yet

   Abstract

   We present CoGS, a novel method for the style-conditioned, sketch-driven synthesis of images. CoGS enables exploration of diverse appearance possibilities for a given sketched object, enabling decoupled control over the structure and the appearance of the output. Coarse-grained control over object structure and appearance are enabled via an input sketch and an exemplar"style"conditioning image to a transformer-based sketch and style encoder to generate a discrete codebook representation. We map the codebook representation into a metric space, enabling fine-grained control over selection and interpolation between multiple synthesis options before generating the image via a vector quantized GA…

3. Close Uses this paper in depth Not relevant

   End-to-End Unsupervised Sketch to Image Generation

   Xingming Lv, Lei Wu, Zhenwei Cheng et al. · 2023 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 1 time · Not summarized yet

   Abstract

   A freehand sketch is geometrically distorted and lacks colors, textures, and other visual details. This leads to challenges in freehand sketch-to-image generation tasks. Many existing works take a multistage strategy to generate images from sketches, and they depend on the shape of the input sketch badly and ignore the shape adjustment of the generated image. More importantly, the generated images are of low quality with distorted textures and colors, and shape deformation. In order to overcome the above challenges, we propose an end-to-end method to accomplish the freehand sketch-to-image task, and the proposed architecture is based on an unsupervised network. Our key insight is to propose …

4. Close Uses this paper in depth Not relevant

   Adversarial Open Domain Adaption Framework (AODA): Sketch-to-Photo Synthesis

   Amey Thakur, M. Satish · 2021 · International Journal of Engineering Applied Sciences and Technology · cited 1 time · Not summarized yet

   Abstract

   This paper aims to demonstrate the efficiency of the Adversarial Open Domain Adaption framework for sketch-to-photo synthesis. The unsupervised open domain adaption for generating realistic photos from a hand-drawn sketch is challenging as there is no such sketch of that class for training data. The absence of learning supervision and the huge domain gap between both the freehand drawing and picture domains make it hard. We present an approach that learns both sketch-to-photo and photo-to-sketch generation to synthesise the missing freehand drawings from pictures. Due to the domain gap between synthetic sketches and genuine ones, the generator trained on false drawings may produce unsatisfac…

5. Close* Uses this paper in depth Not relevant

   SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

   Fang Liu, C. Zou, Xiao-Ming Deng et al. · 2020 · European Conference on Computer Vision · cited 43 times · Not summarized yet