# Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries

**Authors:** Jiaqi Liu, Yonghao Long, Kai Chen, Cheuk Hei Leung, Zerui Wang, Qi Dou  
**arXiv:** [2309.00957](https://arxiv.org/abs/2309.00957)  
**Date:** 2023-09-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.00957v1/

## Key points

- Accurate segmentation of surgical instrument tips is crucial for various applications in robotic surgery
- Existing visual-based methods for segmentation are not practical and suffer from low robustness
- Kinematics data from the robotic system can provide reliable information about instrument location
- The authors propose a novel visual-kinematics graph learning framework for accurate instrument tip segmentation
- The framework incorporates both image and kinematics data to encode relational features of instrument parts
- A graph learning approach is used to capture these features effectively
- A cross-modal contrastive loss function is designed to incorporate robust geometric prior information from kinematics into image-based segmentation
- Experiments were conducted on a private dataset that included multiple procedures such as prostatectomy, total mesorectal excision, fundoplication, distal gastrectomy on cadaver, and distal gastrectomy on porcine
- The proposed multi-modal segmentation method outperforms current state-of-the-art image-based approaches by an average improvement of 11.2% on Dice coefficient

## Summary

Accurate segmentation of surgical instrument tips is crucial for various applications in robotic surgery, including surgical skill assessment, tool-tissue interaction modeling, and surgical autonomy. However, this task is challenging due to the small size of instrument tips and the significant variation in surgical scenes across different procedures. Existing visual-based methods for segmentation suffer from low robustness and are not practical for use. Fortunately, kinematics data from the robotic system can provide reliable information about instrument location that is consistent regardless of the type of surgery. To leverage this multi-modal information, the authors propose a novel visual-kinematics graph learning framework for accurate instrument tip segmentation in different surgical procedures. The proposed framework incorporates both image and kinematics data to encode relational features of instrument parts. A graph learning approach is used to capture these features effectively and a cross-modal contrastive loss function is designed to incorporate robust geometric prior information from kinematics into image-based segmentation. To evaluate their method, experiments were conducted on a private paired visual-kinematics dataset that included multiple procedures such as prostatectomy, total mesorectal excision, fundoplication, distal gastrectomy on cadaver, and distal gastrectomy on porcine. The leave-one-procedure-out cross-validation results demonstrate that the proposed multi-modal segmentation method outperforms current state-of-the-art image based approaches by an average improvement of 11.2% on Dice coefficient. In conclusion, the authors present a novel visual–kinematics graph learning framework for procedure–agnostic instrument tip segmentation in robotic surgeries which effectively utilizes both image and kinematic data to improve robustness and accuracy in segmenting small instrument tips across different surgical procedures. The experimental results highlight the superiority of their approach compared to existing image–based methods.