# Ensembling Instance and Semantic Segmentation for Panoptic Segmentation

**Authors:** Mehmet Yildirim, Yogesh Langhe  
**arXiv:** [2304.10326](https://arxiv.org/abs/2304.10326)  
**Date:** 2023-04-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.10326v1/

## Key points

- Solution for the 2019 COCO panoptic segmentation task
- Approach involves performing instance segmentation and semantic segmentation separately
- Results are combined to generate panoptic segmentation
- Use of expert models of Mask R-CNN to address data imbalance issue in training data for improved instance segmentation performance
- Adoption of HTC model leads to best instance segmentation results
- Multiple models with different backbones trained for semantic segmentation, ensemble strategy employed to enhance results
- Thorough analysis of various combinations of instance and semantic segmentation methods for generating final panoptic segmentation results
- Best model achieves a $PQ$ score of 47.1 on the 2019 COCO panoptic test-dev dataset
- Panoptic segmentation is crucial in computer vision, particularly for applications like autonomous driving
- Annual COCO Panoptic Segmentation Challenge provides opportunity for researchers to showcase solutions and advancements in this field
- Method stands out by addressing challenges such as data imbalance through expert models and leveraging ensemble strategies for improved performance
- Significant progress demonstrated in panoptic segmentation by combining state-of-the-art techniques from both instance and semantic domains

## Summary

In this technical report, we present our solution for the 2019 COCO panoptic segmentation task. Our approach involves performing instance segmentation and semantic segmentation separately and then combining the results to generate panoptic segmentation. To improve the performance of instance segmentation, we incorporate several expert models of Mask R-CNN to address the data imbalance issue in the training data. Additionally, we adopt the HTC model, which leads to our best instance segmentation results. For semantic segmentation, we train multiple models with different backbones and employ an ensemble strategy to further enhance the segmentation results. This combination of techniques allows us to achieve improved performance in semantic segmentation. We thoroughly analyze various combinations of instance and semantic segmentation methods and evaluate their performance for generating final panoptic segmentation results. Our best model achieves a $PQ$ score of 47.1 on the 2019 COCO panoptic test-dev dataset. is a crucial task in computer vision, particularly for applications like autonomous driving. By effectively combining  and   we can achieve a coherent scene understanding that aids in solving real-world problems. The annual COCO Panoptic Segmentation Challenge provides an opportunity for researchers to showcase their solutions and advancements in this field. Our method stands out by addressing challenges such as data imbalance through expert models and leveraging ensemble strategies for improved performance. Overall, our work demonstrates significant progress in panoptic segmentation by combining state-of-the-art techniques from both  and   domains. We believe that our findings contribute valuable insights towards advancing scene understanding tasks like autonomous driving.