Google Scholar
GitHub
LinkedIn
X
Structuring MoE Expert Selection for Agentic Reinforcement Learning
Aligning mixture-of-experts routing with agent operations through hierarchical control during reinforcement-learning post-training.
Sanjoy Chowdhury
Research Scientist at Apple Machine Learning Research
Efficient agents · Reasoning compression · Multimodal learning
firstname_lastname [at] apple [dot] com
Feel free to reach out for research collaborations.
I am a Research Scientist at Apple Machine Learning Research, where I work on efficient on-device tool-calling agents and reasoning compression. My broader research focuses on multimodal foundation models that combine vision, audio, and language for fine-grained perception, grounded reasoning, and cross-modal generation. I am particularly interested in improving their efficiency and reliability through post-training.
I earned my PhD in Computer Science from the University of Maryland, College Park, advised by Prof. Dinesh Manocha, and my MTech in Computer Science & Engineering from IIIT Hyderabad, advised by Prof. C. V. Jawahar.
During my PhD, I worked on audio-visual reasoning and alignment at Apple with Chun-Liang Li and Karren Yang; multisensory egocentric perception at Meta Reality Labs with Ruohan Gao; speech-driven facial synthesis in the Talking Heads team at Google Research with Avisek Lahiri and Vivek Kwatra; and multimodal audio generation at Adobe Research with Joseph K J. I also worked with Mohamed Elhoseiny at KAUST and collaborated with Kristen Grauman and Salman Khan.
Before my PhD, I developed video enhancement algorithms with the Camera and Video AI team at ShareChat and vision systems for consumer devices at Samsung R&D Institute Bangalore. My earlier research included work at ISI Kolkata with Ujjwal Bhattacharya and IIT Kharagpur with Pabitra Mitra.
- Selected as a Top Reviewer at NeurIPS 2026.
- RLTL;DR, our work on learning from self-generated insights, is now on arXiv.
- Updated preprint: Simulate to Generalize, on stateful supervision for API-calling agents.
- Co-organized the second Gen4AVC workshop at ECCV 2026.
More updates
- Co-organized SCALE: Scalable Learning and Optimization for Efficient Multimodal AI Agents at ICML 2026.
- Presented AMUSE at the Apple booth at CVPR 2026.
- New preprint on the representation geometry of minimal reasoning cores.
- Our survey of multimodal agentic frameworks was accepted at TMLR.
- Recognized as a CVPR 2026 Outstanding Reviewer.
- Joined Apple Machine Learning Research as a Research Scientist.
- AMUSE, our benchmark and alignment framework for multi-speaker understanding, was accepted at CVPR 2026.
- Successfully defended my PhD at the University of Maryland, College Park, and announced I would be joining Apple Research as a Research Scientist.
- Recognized as a Top Reviewer by NeurIPS 2025!
- Our paper MAGNET on multi-agent framework for audio-visual RAG got accepted to NeurIPS 2025
- Selected for Doctoral Consortium at ICCV 2025!
- Passed my PhD proposal. Officially a candidate now!
- Co-organising Gen4AVC workshop to be held in conjunction with ICCV 2025.
- Joined Apple MLR as a ML Research intern.
- Invited talk at NYC Computer Vision Day 2025 organised by New York University.
- Invited talk on assessing and addressing the gaps in existing Audio-Visual LLMs at AIR lab at University of Rochester
- Work on Audio-Visual LLM got accepted to ECCV 2024
- Invited talk at the Sight and Sound workshop at CVPR 2024
- Joined Meta Reality Labs as a Research Scientist intern.
- Paper on Improving Robustness Against Spurious Correlations got accepted to ACL 2024 Findings
- Our paper on determining perceived audience intent from multi-modal social media posts got accepted to Nature Scientific Reports
- Paper on LLM guided navigational instruction generation got accepted to NAACL 2024
- MeLFusion (Highlight, Top 2.8%) got accepted to CVPR 2024
- Joined Google Research as a student researcher.
- APoLLo gets accepted to EMNLP 2023
- Invited talk on AdVerb at AV4D Workshop, ICCV 2023
- AdVerb got accepted to ICCV 2023
- Joined Adobe Research as a research intern.
- Joined as a CS PhD student at University of Maryland College Park . Awarded Dean's fellowship.
- Paper on audio-visual summarization accepted in BMVC 2021.
- Blog on Video Quality Enhancement released at Tech @ ShareChat.
- Paper on reflection removal got accepted in ICCV 2021.
- Joined ShareChat Data Science team.
- Paper on audio-visual joint segmentation accepted in ICIP 2021.
- Accepted Samsung Research offer. Will be joining in June'19.
- Received Dean's Merit List Award for academic excellence at IIIT Hyderabad.
- Our work on a multi-scale, low-latency face detection framework received Best Paper Award at NGCT-2017.
Recent publications
View all 22 selected publications ↗2026
RLTL;DR: Self-improvement by Internalizing Self-generated Feedback
Learning from brief, self-generated insights after failed attempts, then internalizing those insights during training.
Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Models
Generating stateful API-calling trajectories with LLM world models to train agents that generalize to unseen environments.
AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
A benchmark for agentic multi-speaker understanding, paired with the RAFT alignment framework.
Presentations: Research presentation · Apple booth, CVPR 2026
Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces
Identifying the compact support within long reasoning traces and studying its representation geometry.
A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
A survey of the techniques, applications, and open challenges in multimodal agentic frameworks.