Publications

2025

WACV 2025
Enhancing vision-language few-shot adaptation with negative learning
Ce Zhang, Simon Stepputtis, Katia Sycara, Yaqi Xie
2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
arXiv 2025
Model-Agnostic Policy Explanations with Large Language Models
Zhang Xi-Jia, Yue Guo, Shufei Chen, Simon Stepputtis, Matthew Gombolay, Katia Sycara, Joseph Campbell
arXiv preprint arXiv:2504.05625
WACV 2025
Negative yields positive: Unified dual-path adapter for vision-language models
Ce Zhang, Simon Stepputtis, Katia Sycara, Yaqi Xie
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
ICLR 2025
OMG: Opacity Matters in Material Modeling with Gaussian Splatting
Silong Yong, Venkata Nagarjun Pudureddiyur Manivannan, Bernhard Kerbl, Zifu Wan, Simon Stepputtis, Katia P. Sycara, Yaqi Xie
The Thirteenth International Conference on Learning Representations
IJCV 2025
Parameter-Efficient Neuro-Symbolic Action Anticipation via Iterative Context Refinement
Fnu Aryan, Yaqi Xie, Joseph Campbel, Woojun Kim, Kwonjoon Lee, Hossein Nourkhiz Mahjoub, Ehsan Moradi-Pari, Katia Sycara,
Under Review at the International Journal of Computer Vision
ICLR 2025
Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q. Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P. Sycara, Yaqi Xie
The Thirteenth International Conference on Learning Representations
WACV 2025
Sigma: Siamese mamba network for multi-modal semantic segmentation
Z Wan, Y Wang, S Yong, P Zhang, S Stepputtis, K Sycara, Y Xie
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
arXiv 2025
Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learning
Muhan Lin, Shuyang Shi, Yue Guo, Vaishnav Tadiparthi, Behdad Chalaki, Ehsan Moradi Pari, Simon Stepputtis, Woojun Kim, Joseph Campbell, Katia Sycara
arXiv preprint arXiv:2502.03723

2024

RA-L 2024
A comparison of imitation learning algorithms for bimanual manipulation
Michael Drolet, Simon Stepputtis, Siva Kailas, Ajinkya Jain, Jan Peters, Stefan Schaal, Heni Ben Amor
IEEE Robotics and Automation Letters
CoLLAs 2024
Adaptive action advising with different rewards
Yue Guo, Xijia Zhang, Simon Stepputtis, Joseph Campbell, Katia Sycara
Conference on Lifelong Learning Agents
NeurIPS 2024
Dual prototype evolving for test-time generalization of vision-language models
Ce Zhang, Simon Stepputtis, Katia Sycara, Yaqi Xie
Advances in Neural Information Processing Systems
ICRA 2024
Geometric Shape Reasoning for Zero-Shot Task-Oriented Grasping
Samuel Li, Sarthak Bhagat, Joseph Campbell, Yaqi Xie, Woojun Kim, Katia P. Sycara, Simon Stepputtis
ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation
NeurIPS 2024
Gl-nerf: Gauss-laguerre quadrature enables training-free nerf acceleration
Silong Yong, Yaqi Xie, Simon Stepputtis, Katia Sycara
Advances in Neural Information Processing Systems
arXiv 2024
HiMemFormer: Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipation
Zirui Wang, Xinran Zhao, Simon Stepputtis, Woojun Kim, Tongshuang Wu, Katia Sycara, Yaqi Xie
arXiv preprint arXiv:2411.01455
CVPR 2024
Hiker-sgg: Hierarchical knowledge enhanced robust scene graph generation
Ce Zhang, Simon Stepputtis, Joseph Campbell, Katia Sycara, Yaqi Xie
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
NeurIPS 2024
Incorporating Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q. Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P. Sycara, Yaqi Xie
NeurIPS 2024 Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models
AAAI 2024
Instructpart: Affordance-based part segmentation from language instruction
Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, Katia P Sycara
AAAI-2024 Workshop on Public Sector LLMs: Algorithmic and Sociotechnical Design
RA-L 2024
Let Me Help You! Neuro-Symbolic Short-Context Action Anticipation
Sarthak Bhagat, Samuel Li, Joseph Campbell, Yaqi Xie, Katia Sycara, Simon Stepputtis
IEEE Robotics and Automation Letters
NeurIPS 2024
LogiCity: Advancing neuro-symbolic ai with abstract urban simulation
Bowen Li, Zhaoyu Li, Qiwei Du, Jinqi Luo, Wenshan Wang, Yaqi Xie, Simon Stepputtis, Chen Wang, Katia Sycara, Pradeep Ravikumar, others
Advances in Neural Information Processing Systems
ICRA 2024
Multi-Agent Transfer Learning via Temporal Contrastive Learning
Weihao Zeng, Joseph Campbell, Simon Stepputtis, Katia Sycara
ICRA 2024 Workshop on MAD-GAMES: MULTI-AGENT DYNAMIC GAMES
arXiv 2024
Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
Muhan Lin, Shuyang Shi, Yue Guo, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Simon Stepputtis, Joseph Campbell, Katia Sycara
arXiv preprint arXiv:2410.17389
IROS 2024
Shapegrasp: Zero-shot task-oriented grasping with large language models through geometric decomposition
Samuel Li, Sarthak Bhagat, Joseph Campbell, Yaqi Xie, Woojun Kim, Katia Sycara, Simon Stepputtis
2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
arXiv 2024
Symbolic Graph Inference for Compound Scene Understanding
FNU Aryan, Simon Stepputtis, Sarthak Bhagat, Joseph Campbell, Kwonjoon Lee, Hossein Nourkhiz Mahjoub, Katia Sycara
arXiv preprint arXiv:2410.22626
ICML 2024
Test-Time Prototype Evolving for Generalizable Vision-Language Models
Ce Zhang, Simon Stepputtis, Katia P. Sycara, Yaqi Xie
ICML 2024 Workshop on Foundation Models in the Wild

2023

SMC 2023
A Framework for Intervention Based Team Support in Time Critical Tasks
Dana Hughes, Huao Li, Max Chis, Ini Oguntola, Simon Stepputtis, Keyang Zheng, Joseph Campbell, Katia Sycara, Michael Lewis
2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
arXiv 2023
Benchmarking and enhancing disentanglement in concept-residual models
Renos Zabounidis, Ini Oguntola, Konghao Zhao, Joseph Campbell, Simon Stepputtis, Katia Sycara
arXiv preprint arXiv:2312.00192
NeurIPS 2023
Characterizing out-of-distribution error via optimal transport
Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepputtis, Joseph Campbell, Katia Sycara
Advances in Neural Information Processing Systems
CoRL 2023
Concept learning for interpretable multi-agent reinforcement learning
Renos Zabounidis, Joseph Campbell, Simon Stepputtis, Dana Hughes, Katia P Sycara
Conference on Robot Learning
ICRA 2023
Explainable action advising for multi-agent reinforcement learning
Yue Guo, Joseph Campbell, Simon Stepputtis, Ruiyu Li, Dana Hughes, Fei Fang, Katia Sycara
2023 IEEE International Conference on Robotics and Automation (ICRA)
IROS 2023
Explaining agent behavior with large language models
Xijia Zhang, Yue Guo, Simon Stepputtis, Katia Sycara, Joseph Campbell
Human Multi-Robot Interaction Workshop at IROS 2023
CoLLAs 2023
Introspective action advising for interpretable transfer learning
Joseph Campbell, Yue Guo, Fiona Xie, Simon Stepputtis, Katia Sycara
Conference on Lifelong Learning Agents
ICCV 2023
Knowledge-guided short-context action anticipation in human-centric videos
Sarthak Bhagat, Simon Stepputtis, Joseph Campbell, Katia Sycara
ICCV 2023 Workshop on AI for Creative Video Editing and Understanding
AuRo 2023
Learning modular language-conditioned robot policies through attention
Yifan Zhou, Shubham Sonawani, Mariano Phielipp, Heni Ben Amor, Simon Stepputtis
Autonomous Robots
EMNLP 2023
Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models
Simon and Campbell, Joseph and Xie, Yaqi and Qi, Zhengyang and Zhang, Wenxin and Wang, Ruiyi and Rangreji, Sanketh and Lewis, Charles and Sycara, Katia Stepputtis
Findings of the Association for Computational Linguistics: EMNLP 2023
NeurIPS 2023
Robust Hierarchical Scene Graph Generation
Ce Zhang, Simon Stepputtis, Joseph Campbell, Katia Sycara, Yaqi Xie
NeurIPS 2023 Workshop: New Frontiers in Graph Learning
CoLLAs 2023
Sample-efficient learning of novel visual concepts
Sarthak Bhagat, Simon Stepputtis, Joseph Campbell, Katia Sycara
Conference on Lifelong Learning Agents
EMNLP 2023
Theory of Mind for Multi-Agent Collaboration via Large Language Models
Huao and Chong, Yu and Stepputtis, Simon and Campbell, Joseph and Hughes, Dana and Lewis, Charles and Sycara, Katia Li
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
ICML 2023
Theory of mind as intrinsic motivation for multi-agent reinforcement learning
Ini Oguntola, Joseph Campbell, Simon Stepputtis, Katia Sycara
ICML 2023 Workshop on Theory of Mind
arXiv 2023
Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation
Xijia Zhang, Yue Guo, Simon Stepputtis, Katia Sycara, Joseph Campbell
arXiv preprint arXiv:2311.18062

2022

IROS 2022
A system for imitation learning of contact-rich bimanual manipulation policies
Simon Stepputtis, Maryam Bandari, Stefan Schaal, Heni Ben Amor
2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
CoRL 2022
Modularity through attention: Efficient training and transfer of language-conditioned policies for robot manipulation
Yifan Zhou, Shubham Sonawani, Mariano Phielipp, Simon Stepputtis, Heni Ben Amor
Conference on Robot Learning (CoRL)

2020

NeurIPS 2020
Language-conditioned imitation learning for robot manipulation tasks
Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, Heni Ben Amor
Advances in Neural Information Processing Systems

2019

NeurIPS 2019
Imitation learning of robot policies by combining language, vision and demonstration
Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Chitta Baral, Heni Ben Amor
NeurIPS 2019 Workshop on Robot Learning: Control and Interaction in the Real World
IROS 2019
Improved exploration through latent trajectory optimization in deep deterministic policy gradient
Kevin Sebastian Luck, Mel Vecerik, Simon Stepputtis, Heni Ben Amor, Jonathan Scholz
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
IROS 2019
Learning interactive behaviors for musculoskeletal robots using bayesian interaction primitives
Joseph Campbell, Arne Hitzmann, Simon Stepputtis, Shuhei Ikemoto, Koh Hosoda, Heni Ben Amor
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
RSS 2019
Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks
Joseph Campbell, Simon Stepputtis, Heni Ben Amor
Robotics: Science and Systems XV

2018

ICRA 2018
Extrinsic dexterity through active slip control using deep predictive models
Simon Stepputtis, Yezhou Yang, Heni Ben Amor
2018 IEEE International Conference on Robotics and Automation (ICRA)
AuRo 2018
One-shot learning of human--robot handovers with triadic interaction meshes
David Vogt, Simon Stepputtis, Bernhard Jung, Heni Ben Amor
Autonomous Robots

2017

ICRA 2017
A system for learning continuous human-robot interactions from human-human demonstrations
David Vogt, Simon Stepputtis, Steve Grehl, Bernhard Jung, Heni Ben Amor
2017 IEEE International Conference on Robotics and Automation (ICRA)
RSS 2017
Active Slip Control for In-Hand Object Manipulation using Deep Predictive Models
Simon Stepputtis, Heni Ben Amor
Workshop on Tactile Sensing for Manipulation: Hardware, Modeling, and Learning (RSS)
RSS 2017
Deep Predictive Models for Active Slip Control
Simon Stepputtis, Heni Ben Amor
Workshop on (Empirically) Data-Driven Robotic Manipulation (RSS)

2016

Humanoids 2016
Learning human-robot interactions from human-human demonstrations (with applications in lego rocket assembly)
David Vogt, Simon Stepputtis, Richard Weinhold, Bernhard Jung, Heni Ben Amor
2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)