TrajPred

Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models — IROS 2026

TrajPred encodes instrument trajectories to inject temporal motion cues into Vision-Language Models, then uses a trajectory-conditioned predictor to generate visual semantic embeddings that capture fine-grained action details. With prompt tuning and a verb-rephrasing technique for task adaptation, TrajPred improves instrument–tissue interaction recognition on CholecT50 in both Average Precision and Top-K accuracy.

---

Abstract

Recognizing instruments' interactions with tissues is essential for building context-aware AI assistants in robotic surgery. Vision-language models (VLMs) have opened a new avenue for surgical perception and achieved better generalization on a wide range of tasks compared to conventional task-specific deep learning approaches. However, their performance on instrument–tissue interaction recognition remains limited, largely due to two challenges: (1) many models do not effectively leverage temporal information, and (2) alignment between vision and text often misses fine-grained action details. To address these issues, we propose TrajPred, a framework that encodes instrument trajectories to incorporate temporal motion cues; and conditioned on these trajectories, introduces a predictor module to generate visual semantic embeddings that better capture fine-grained action details. We further incorporate prompt tuning and a verb-rephrasing technique to enable smooth adaptation to the instrument–tissue interaction recognition task. Extensive experiments on the public laparoscopic benchmark, CholecT50, show that our method improves both Average Precision and Top-K accuracy.

We also investigate whether visual embeddings of instrument–tissue interaction regions align better with the corresponding text by visualizing the cosine similarity between visual and textual embeddings. The visualization results indicate that the proposed method improves alignment between relevant visual and textual representations.

---

Figures

---

Citation

@article{cheng2026trajpred,
  title={TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models},
  author={Cheng, Jiajun and Yu, Xiaofan and Tripathi, Subarna and Liu, Sainan and Lin, Shan},
  journal={arXiv preprint arXiv:2603.06999},
  year={2026}
}

---

Authors

---

Update — Metrics Bug Fix

We identified a bug in our metrics implementation that only affects the CLIP-VIT-L-Patch14-FT (vision encoder frozen) row of Table 2 in the paper. The fix does not change the paper's main claims.