Projects

Current Projects

View all publications →
project thumbnail

Surgical Workflow Modeling

Vision-Language • Trustworthy AI • Skill Assessment
project thumbnail

Surgical Navigation

Coming Soon
project thumbnail

Digital Twin

3D Perception • Implicit Neural Representations • Registration
project thumbnail

Robotic Manipulation

Soft Object • Real-to-Sim • Sim-to-Real
project thumbnail

Multimodal Sensing

Force • Tactile • Thermal

Past Projects

project thumbnail

Robot Pose Tracking

Occlusion • Calibration • In-the-Wild
project thumbnail

Semantic Segmentation

Data Efficiency • Robust Features • Real-Time

All Publications

2026

  1. OphIn-500K.png
    OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models
    Xuanzhao Dong, Wenhui Zhu, Xiwen Chen, Hao Wang, Xin Li, Yujian Xiong, Jiajun Cheng, Jingjing Wang, Xiaobing Yu, Haiyu Wu, and others
    arXiv preprint arXiv:2605.27916, 2026
    Paper
  2. trajpred.png
    TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models
    Jiajun Cheng, Xiaofan Yu, Subarna Tripathi, Sainan Liu, and Shan Lin
    IROS 2026
    Paper
  3. twin.png
    TwinTrack: Bridging Vision and Contact Physics for Real-Time Tracking of Unknown Dynamic Objects
    Wen Yang, Zhixian Xie, Xuechao Zhang, Heni Ben Amor, Shan Lin, and Wanxin Jin
    ICRA 2026
    Project Paper Video
  4. palpad.png
    PalpAid: Multimodal Pneumatic Tactile Sensor for Tissue Palpation
    Devi Yuliarti, Ravi Prakash, Hiu Ching Cheung, Amy Strong, Patrick J Codd, and Shan Lin
    RoboSoft 2026
    Paper Video Code
  5. kidney_stone.png
    Detection of fragmentation while dusting during retrograde intrarenal laser lithotripsy: a novel computer vision and perception pipeline
    Jonathan E Katz, Orlando Diaz-Ramos, Christopher Yong-Zyn Lo, Jamie Finegan, Tung Yen Chiang, Yijie He, Zekai Liang, Michael Yip, Roger L Sur, and Shan Lin
    Lasers in medical science, 2026
    Paper
  6. causal2.png
    Surgxbench: Explainable vision-language model benchmark for surgery
    Jiajun Cheng, Xianwu Zhao, Sainan Liu, Xiaofan Yu, Ravi Prakash, Patrick J Codd, Jonathan Elliott Katz, and Shan Lin
    WACV 2026
    Project Paper Code

2025

  1. nerf_cystoscopy.png
    3D Rendering of Cystoscopy Video Footage: A Novel Method Utilizing Neural Radiance Field Processing
    Jonathan E Katz, Jamie Finegan, Pablo F Beutelspacher, Jingpei Lu, Shan Lin, Michael Yip, Roger L Sur, Jamie L Finegan, and Roger Sur
    Cureus, 2025
    Paper
  2. based.jpg
    BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction Using Neural Radiance Fields
    Shreya Saha, Zekai Liang, Shan Lin, Jingpei Lu, Michael Yip, and Sainan Liu
    WACV 2025
    Paper Video
  3. ctrnet.png
    CtRNet-X: Camera-to-Robot Pose Estimation in Real-World Conditions Using a Single Camera
    Jingpei Lu, Zekai Liang, Tristin Xie, Florian Ritcher, Shan Lin, Sainan Liu, and Michael C Yip
    ICRA 2025
    Paper Video

2024

  1. fig_system_overview.png
    Efficient Data-Driven Joint-Level Calibration of Cable-Driven Surgical Robots
    Haonan Peng, Andrew Lewis, Yun-Hsuan Su, Shan Lin, Dun-Tin Chiang, Wenfan Jiang, Helen Lai, and Blake Hannaford
    npj Robotics, 2024
    Paper
  2. superpm.png
    SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data
    Shan Lin, Albert J Miao, Ali Alabiad, Fei Liu, Kaiyuan Wang, Jingpei Lu, Florian Richter, and Michael C Yip
    IROS 2024
    Paper Video
  3. reduce_load_haonan_MIA2024.png
    Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation
    Haonan Peng, Shan Lin, Daniel King, Yun-Hsuan Su, Waleed M Abuzeid, Randall A Bly, Kris S Moe, and Blake Hannaford
    Medical Image Analysis, 2024
    Paper
  4. hemoset.png
    HemoSet: The First Blood Segmentation Dataset for Automation of Hemostasis Management
    Albert J Miao, Shan Lin, Jingpei Lu, Florian Richter, Benjamin Ostrander, Emily K Funk, Ryan K Orosco, and Michael C Yip
    ISMR 2024 . 🏆 Best Student Paper Award
    Paper Data
  5. openx.png
    Open X-Embodiment: Robotic Learning Datasets and RT-X Models
    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, and others
    ICRA 2024 . 🏆 Best Conference Paper Award
    Project Paper
  6. qualitative_lab.png
    Tracking Snake-Like Robots in the Wild Using Only a Single Camera
    Jingpei Lu, Florian Richter, Shan Lin, and Michael C Yip
    ICRA 2024
    Paper Video
  7. one-shot-cover.jpg
    AnyOKP: One-Shot and Instance-Aware Object Keypoint Extraction with Pretrained ViT
    Fangbo Qin, Taogang Hou, Shan Lin, Kaiyuan Wang, Michael C Yip, and Shan Yu
    ICRA 2024
    Paper Video Code
  8. real_to_sim.png
    Real-to-Sim Deformable Object Manipulation: Optimizing Physics Models with Residual Mappings for Robotic Surgery
    Xiao Liang, Fei Liu, Yutong Zhang, Yuelei Li, Shan Lin, and Michael Yip
    ICRA 2024
    Paper Video
  9. droid.png
    DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset
    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, and others
    RSS 2024
    Project Paper
  10. baangp.png
    BAA-NGP: Bundle-Adjusting Accelerated Neural Graphics Primitives
    Sainan Liu, Shan Lin, Jingpei Lu, Alexey Supikov, and Michael Yip
    CVPR 2024 Workshop on Visual Odometry and Computer Vision Applications Based on Location Clues
    Paper Code

2023

  1. orrn.png
    ORRN: An ODE-Based Recursive Registration Network for Deformable Respiratory Motion Estimation with Lung 4DCT Images
    Xiao Liang, Shan Lin, Fei Liu, Dimitri Schreiber, and Michael Yip
    IEEE Transactions on Biomedical Engineering, 2023
    Paper Video Code
  2. semanticsuper.png
    Semantic-Super: A Semantic-Aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking
    Shan Lin, Albert J Miao, Jingpei Lu, Shunkai Yu, Zih-Yun Chiu, Florian Richter, and Michael C Yip
    ICRA 2023
    IROS 2023 Workshop on Integrated Perception, Planning, and Control for Physically and Contextually-Aware Robot Autonomy
    Paper Video Code

2022

  1. contour.png
    Contour Primitive of Interest Extraction Network Based on Dual-Metric One-Hot Learning for Vision Measurement
    Fangbo Qin, Shan Lin, and De Xu
    IEEE Transactions on Industrial Informatics, 2022
    Paper
  2. zixin.png
    Endoscope Localization and Dense Surgical Scene Reconstruction for Stereo Endoscopy by Unsupervised Optical Flow and Kanade-Lucas-Tomasi Tracking
    Zixin Yang, Shan Lin, Richard Simon, and Cristian A Linte
    EMBC 2022
    Paper Code

2021

  1. MFFA.png
    Multi-Frame Feature Aggregation for Real-Time Instrument Segmentation in Endoscopic Video
    Shan Lin, Fangbo Qin, Haonan Peng, Randall A Bly, Kris S Moe, and Blake Hannaford
    IEEE Robotics and Automation Letters, 2021
    Paper

2020

  1. LCGAN.png
    LC-GAN: Image-to-Image Translation Based on Generative Adversarial Network for Endoscopic Images
    Shan Lin, Fangbo Qin, Yangming Li, Randall A Bly, Kris S Moe, and Blake Hannaford
    IROS 2020
    Paper
  2. MAFA.jpg
    Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision
    Fangbo Qin, Shan Lin, Yangming Li, Randall A Bly, Kris S Moe, and Blake Hannaford
    IEEE Robotics and Automation Letters, 2020
    Paper
  3. spieskill.png
    Video-Based Automatic and Objective Endoscopic Sinus Surgery Skill Assessment
    Shan Lin, Xinyu Gu, Randall A Bly, Kris S Moe, and Blake Hannaford
    Medical Imaging 2020: Image-Guided Procedures, Robotic Interventions, and Modeling
    Paper PDF

2019

  1. mmmi.png
    Automatic Sinus Surgery Skill Assessment Based on Instrument Segmentation and Tracking in Endoscopic Video
    Shan Lin, Fangbo Qin, Randall A Bly, Kris S Moe, and Blake Hannaford
    MICCAI 2019 International Workshop on Multiscale Multimodal Medical Imaging
    Paper

2017

  1. burn.png
    Don’t Get Burned: Thermal Monitoring of Vessel Sealing Using a Miniature Infrared Camera
    Shan Lin, Loris Fichera, Mitchell J Fulton, and Robert J Webster III
    Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling
    Paper PDF