Monocular Pose Estimation of Articulated Open Surgery Tools — In the Wild

Robert Spektor1, Tom Friedman2, Itay Or2, Gil Bolotin2, Shlomi Laufer1
1Faculty of Data and Decision Sciences, Technion – Israel Institute of Technology, Haifa, Israel
2Rambam Health Care Campus, Haifa, Israel
Medical Image Analysis, 2025
Paper (MedIA) arXiv Code (coming soon) Dataset

Abstract

This work presents a framework for monocular 6D pose estimation of surgical instruments in open surgery, addressing challenges such as object articulations, specularity, occlusions, and synthetic-to-real domain adaptation. The proposed approach consists of three main components: (1) a synthetic data generation pipeline that incorporates 3D scanning of surgical tools with articulation rigging and physically-based rendering; (2) a tailored pose estimation framework combining tool detection with pose and articulation estimation; and (3) a training strategy on synthetic and real unannotated video data, employing domain adaptation with automatically generated pseudo-labels.

Evaluations conducted on real data of open surgery demonstrate the good performance and real-world applicability of the proposed framework, highlighting its potential for integration into medical augmented reality and robotic systems. The approach eliminates the need for extensive manual annotation of real surgical data.

Method

Pose estimation model framework pipeline

An input RGB image is cropped to the Region of Interest and processed by a Multitask Feature Regression Network (ViT backbone + UPerNet/PPM) that predicts object category, articulation angle, object masks, and a dense 2D–3D correspondence map. The correspondences feed a Patch-PnP module (training) or PnP-RANSAC (inference) to recover the 6D pose.

Dataset

We release the resources used in this work. The real-world data was collected from ten Coronary Artery Bypass Graft (CABG) vein-harvesting procedures, focusing on the suturing phase, under approval of the Rambam Health Care Campus IRB committee. We provide instance segmentation masks of surgical tools (needle-holder, tweezers) and hands, alongside the 3D tool meshes used to generate synthetic training data.

Annotated segmentation masks of surgical tools and hands
3D meshes of surgical tools in varying degrees of articulation

Surgical tool meshes shown across varying degrees of articulation.

Access is granted upon request. Submit your email through the form below and you will automatically receive a download link and password. By requesting access you agree to use the data for non-commercial research only, in accordance with the CC BY-NC 4.0 license.

Request Access

BibTeX

@article{spektor2025monocular,
  title   = {Monocular pose estimation of articulated open surgery tools - in the wild},
  author  = {Spektor, Robert and Friedman, Tom and Or, Itay and Bolotin, Gil and Laufer, Shlomi},
  journal = {Medical Image Analysis},
  year    = {2025},
  doi     = {10.1016/j.media.2025.103618},
  url     = {https://doi.org/10.1016/j.media.2025.103618}
}