LAViF

Latent-Action-Guided Vision–Language Contrastive Learning for Surgical Interaction Recognition

Jiajun Cheng¹, Sainan Liu³, Subarna Tripathi³, Xiaofan Yu², Shan Lin¹

¹Arizona State University · ²University of California, Merced · ³Intel Corporation

📄 Paper (arXiv) 💻 Code — coming soon

The current arXiv posting is an earlier version of this work, titled LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition. An updated version is in preparation.

PCA-RGB maps comparing ViT-L, V-JEPA2, V-JEPA2.1 and LAViF — PCA-RGB shows the top three principal components of patch features as RGB. LAViF adapts a plain ViT-L into spatially coherent maps, with clearer emphasis on the interacting instrument and target tissue.

Abstract

Pretrained encoders often lack spatial coherence, and global vision–language alignment does not guarantee precise spatial or temporal structure. LAViF compresses frame-to-frame feature changes into latent actions and predicts next-frame features during end-to-end video–language alignment. Without extra spatial or motion annotations it improves interaction grounding and temporal-direction sensitivity, and with ordinary image encoders reaches competitive recognition at higher throughput and smaller INT4 drops than V-JEPA2/2.1.

Contributions

Semantic alignment reshapes visual transition structure — and more dimensional expansion does not mean better recognition.
LAViF shapes that structure with a Causal Transition IDM, a compact predictive action bottleneck, and patch-level SIGReg.
Across three encoders and two datasets: better grounding without extra supervision, faster inference, smaller INT4 drops.

Method

LAViF pipeline: vision encoder, IDM+FDM training, vision-language alignment — Frames are encoded independently and pooled into per-frame states, with patch-level SIGReg stabilizing local features. A Causal Transition **IDM** compresses consecutive-state differences into an action token that conditions a DiT **forward dynamics model (FDM)**; states and actions are aggregated and aligned to triplet text.

Results

Main comparison on CholecT50 (Rendezvous split) and ProstaTD (5-fold) with latent-action dimension *d_a* = 128. ^† frame-wise image encoders, ^‡ native video encoders, ^△ classification head without video–language alignment, ^◇ evaluated with different metrics and therefore requiring cautious comparison. **Bold** and underline mark best and second-best.
Method	mAP_IVT	mAP_I	mAP_V	mAP_T	mAP_IVT	mAP_I	mAP_V	mAP_T
	CholecT50 (RDV)				ProstaTD (5-fold)
Published task-specific methods
Rendezvous^†△	29.90	92.20	60.07	38.30	–	–	–	–
Chain-of-Look^†△	38.00	94.10	62.50	41.90	–	–	–	–
MML-SurgAdapt^†	30.30	87.30	61.10	43.20	–	–	–	–
TDNet^†◇△	–	–	–	–	36.10	89.90	61.70	55.70
Controlled baselines — general-domain pretraining
CLIP ViT-L/14^†	26.17	86.26	62.54	42.93	26.80	90.97	65.00	53.51
DINOv2-L^†	27.41	90.08	65.28	45.44	36.01	93.13	70.75	63.84
V-JEPA2^‡	29.68	92.16	68.55	47.97	32.66	91.30	71.89	59.95
V-JEPA2.1^‡	34.82	93.99	69.76	50.23	35.87	94.36	74.35	62.64
TimeSformer^‡	27.32	88.67	64.69	44.43	29.43	86.19	64.69	56.14
Controlled baselines — surgical-domain pretraining
EndoViT^†	24.71	85.19	61.41	41.23	26.94	88.35	62.02	53.63
LEMON^†	28.71	87.34	65.04	44.70	32.26	91.72	66.51	59.71
SurgVLP^†	20.28	73.96	52.33	37.76	20.64	78.07	52.45	45.88
HecVLP^†	21.41	77.59	57.23	39.29	22.42	81.16	55.89	48.76
PeskaVLP^†	22.02	79.20	57.22	39.26	23.09	82.45	56.99	50.75
LAViF — ViT-L	32.15	93.24	68.29	47.05	32.82	92.82	68.28	60.63
LAViF — DINOv2-L	33.13	94.60	68.91	48.81	38.14	96.03	73.01	65.21
LAViF — SurgeNet-L	32.78	95.02	71.64	50.89	38.85	96.74	72.82	65.73

Best instrument, verb and target mAP on CholecT50, and best triplet, instrument and target mAP on ProstaTD — from image encoders without large-scale video pretraining. Chain-of-Look still leads CholecT50 triplet mAP, using a task-specific classification head. ProstaTD values are five-fold means.

Spatial feature organization

PCA-RGB maps comparing LAViF, no-IDM+FDM baselines and V-JEPA2/2.1 over time — PCA-RGB over a frame sequence: full LAViF against the same encoders without IDM+FDM, and against finetuned V-JEPA2/2.1.

Prediction weight sweep — drag the slider

ViT-L/16

DINOv2-L/16

SurgeNet-L/14

λ_pred swept at fixed d_a = 128. Predictive pressure reorganizes local structure for every encoder, but unevenly: going 0 → 1 gains ViT-L 2.81 verb / 1.40 triplet mAP and SurgeNet-L 1.75 verb / 1.51 target mAP, while costing SurgeNet-L 0.79 triplet mAP.

Show the full published grid (all encoders)

Full PCA-RGB grid across prediction weights — PCA-RGB across λ_pred; the red ellipse marks the instrument–target interaction region.

PCA-RGB ablation across three encoders: full model, without patch-level SIGReg, and without IDM+FDM — Full LAViF (*d_a* = 128) against no-patch-SIGReg and no-IDM+FDM, for ViT-L/16, DINOv2-L/16 and SurgeNet-L/14.

Deployment efficiency

LAViF encodes frames independently, so attention cost is linear in clip length, O(T·P² + T²), against O(T²·P²) for spatiotemporal encoders — the gap widens with longer clips.

Inference efficiency on CholecT50 (single GPU, real clips). Throughput is the headline metric; latency is per clip, single stream. ↑ / ↓ = higher / lower is better.
Model	Attention	Res.	Params	Latency (ms) ↓	Throughput (clips/s) ↑	Memory (GB) ↓
Ours (per-frame)
LAViF — ViT-L	per-frame	224	304M	13.7	72.8	7.5
LAViF — DINOv2-L	per-frame	224	304M	19.7	50.8	10.9
LAViF — SurgeNet-L	per-frame	224	304M	41.5	24.1	20.2
Video baselines (spatiotemporal)
V-JEPA2	spatiotemporal	256	326M	68.1	14.7	15.4
V-JEPA2.1	spatiotemporal	384	305M	96.2	10.4	22.0
TimeSformer	spatiotemporal	384	305M	94.3	10.6	22.0

At ~300M parameters, ViT-L and DINOv2-L reach 50–73 clips/s, about 3.5–5× the spatiotemporal baselines (≤ 14.7). SurgeNet-L is slower at 24.1 clips/s, still above every video baseline.

Relative mAP change (%) under post-training, weight-only INT4 quantization on CholecT50, against FP32.
Encoder	Variant	ΔI	ΔV	ΔT	ΔIVT
ViT-L	Ours	−0.9	−0.8	0.0	−2.7
ViT-L	No IDM+FDM	−0.1	−0.5	−2.9	−4.2
DINOv2-L	Ours	−0.3	−0.2	−0.1	−2.8
DINOv2-L	No IDM+FDM	−0.5	−0.9	−0.9	−4.2
SurgeNet-L	Ours	−0.1	−0.6	+0.2	+0.4
SurgeNet-L	No IDM+FDM	−0.2	−1.0	−0.5	−1.3
V-JEPA2	Base	+0.2	−2.4	−3.9	−7.7
V-JEPA2.1	Base	−2.4	−4.8	−4.7	−5.6

Citation

@article{cheng2026lavifit,
  title={Latent-Action-Guided Vision-Language Contrastive Learning for Surgical Interaction Recognition},
  author={Cheng, Jiajun and Liu, Sainan and Tripathi, Subarna and Yu, Xiaofan and Lin, Shan},
  journal={arXiv preprint arXiv:2607.19889},
  year={2026}
}

Authors

Jiajun Cheng

ASU

Sainan Liu

Intel

Subarna Tripathi

Intel

Xiaofan Yu

University of California, Merced

Shan Lin

ASU