Agents & Efficient Reasoning
2025–present
Research garden
My work across six research directions. Each flower represents a paper; the stems group related topics, with recent work toward the top.
Select a flower to explore a paper
22 selected papers · 2021–2026Cross-modal Generation
2023–present
Audio-Visual Representation Learning
2021–present
Audio-Visual LLMs
2024–present
Integrating Vision-Language
2022–present
Computational Photography
2021–2023
Newer papers appear toward the top of each stem. The full descriptions and paper lists follow below.
Explore by research direction
View publications by year ↗Research direction
Agents & Efficient Reasoning
2025–present
At Apple, my current focus is efficient on-device tool-calling agents and reasoning compression. My public work studies how agents learn from feedback, reason over evidence, and act in unfamiliar environments. MAGNET (NeurIPS 2025) coordinates agents across multiple videos. In 2026, Simulate to Generalize scales stateful API-calling supervision with LLM world models; RLTL;DR internalizes self-generated insights; and MoE Expert Selection aligns expert routing with agent operations. Minimal Cores examines which reasoning steps support a prediction, while our TMLR survey maps perception, reasoning, planning, memory, and action in multimodal agents.
Structuring MoE Expert Selection for Agentic Reinforcement Learning
Aligning mixture-of-experts routing with agent operations through hierarchical control during reinforcement-learning post-training.
RLTL;DR: Self-improvement by Internalizing Self-generated Feedback
Learning from brief, self-generated insights after failed attempts, then internalizing those insights during training.
Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Models
Generating stateful API-calling trajectories with LLM world models to train agents that generalize to unseen environments.
Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces
Identifying the compact support within long reasoning traces and studying its representation geometry.
A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
A survey of the techniques, applications, and open challenges in multimodal agentic frameworks.
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
Multi-agent retrieval and reasoning over audio-visual evidence distributed across multiple videos.
Research direction
Cross-modal Generation
2023–present
How can visual and language cues guide the generation or refinement of audio? MeLFusion (CVPR 2024, Highlight · Top 2.8%) combines images and text to synthesize music with diffusion models. AdVerb (ICCV 2023) uses visual context to estimate clean audio from reverberant recordings. Together, they study cross-modal conditioning for creative synthesis and audio restoration. More recent work, including MAGNET and AMUSE, broadens my focus to evidence-grounded multimodal reasoning and agents; those papers appear in the related directions below.
MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models
Synthesizing music from image and text cues with a diffusion model and visual synapses.
Presentations: Invited talk · Sight and Sound, CVPR 2024
AdVerb: Visually Guided Audio Dereverberation
Using visual context to help estimate clean audio from reverberant recordings.
Presentations: Invited talk · AV4D, ICCV 2023
Research direction
Audio-Visual Representation Learning
2021–present
This direction develops useful audio-visual representations while controlling the cost of perception. Listen to the Pixels (ICIP 2021) learns joint segmentation from correspondences between sound and vision. AudViSum (BMVC 2021) uses self-supervised reinforcement learning to produce diverse audio-visual summaries. EgoAdapt (ICCV 2025) brings this strand to efficient egocentric perception through adaptive cross-modal distillation and policy learning, covering action recognition, active-speaker localization, and behavior anticipation. This interest in selecting useful information also connects to my newer work on efficient reasoning.
EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception
Adaptive multisensory distillation and policy learning for efficient egocentric perception.
Presentations: Oral presentation · BinEgo-360°, ICCV 2025 Invited talk · NYC Computer Vision Day, NYU 2025
AudViSum: Self-Supervised Deep Reinforcement Learning for Diverse Audio-Visual Summary Generation
Self-supervised reinforcement learning for diverse, informative audio-visual summaries.
Listen to the Pixels
Self-supervised joint audio-visual segmentation by exploiting correspondence between the modalities.
Research direction
Audio-Visual LLMs
2024–present
I build and evaluate models that jointly ground sound, video, and language. Meerkat (ECCV 2024) addresses fine-grained grounding in space and time. AURELIA (ICCV 2025) improves structured reasoning at test time, while AVTrustBench (ICCV 2025) examines robustness and reliability. AURA separates perceptual grounding from the validity of reasoning steps, revealing gaps that answer accuracy can miss. AMUSE (CVPR 2026) extends the direction to multi-speaker interactions: identifying who speaks, tracking roles and events, and evaluating agentic understanding, with RAFT for data-efficient alignment.
AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
A benchmark for agentic multi-speaker understanding, paired with the RAFT alignment framework.
Presentations: Research presentation · Apple booth, CVPR 2026
AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning
Evaluating audio-visual reasoning traces by separating factual consistency from the validity of core inferences.
AURELIA: Test-time Reasoning Distillation in Audio-Visual LLMs
An actor–critic framework for structured audio-visual reasoning at test time, alongside AVReasonBench.
Presentations: Invited presentation · CLVL, ICCV 2025 Invited presentation · AVGenL, ICCV 2025
AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs
Evaluating audio-visual reliability under adversarial attacks, compositional reasoning, and modality-specific dependencies.
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time
Fine-grained audio-visual grounding in space and time through modality alignment and instruction tuning.
Presentations: Invited talk · AVGenL, ECCV 2024
Research direction
Integrating Vision-Language
2022–present
My vision-language work connects robust representation learning to understanding and embodied interaction. APoLLo (EMNLP 2023) combines adapters and prompts for few-shot adaptation. ASPIRE (ACL Findings 2024) uses language-guided augmentation to reduce reliance on spurious visual cues. Wayfinding / VLMNav (NAACL 2024) synthesizes platform-independent navigation instructions through in-context learning. Intent-o-meter (Scientific Reports 2024) combines image, text, and psychological cues to model perceived intent in social media posts. These foundations connect to the newer agentic work on learning, reasoning, and acting across environments.
Towards Determining Perceived Human Intent for Multimodal Social Media Posts using The Theory of Reasoned Action
Predicting perceived intent in multimodal social media posts using visual, textual, and psychological cues.
ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations
Language-guided image augmentation to reduce dependence on spurious visual features.
Can LLM’s Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis
Generating platform-independent wayfinding instructions with in-context learning.
APoLLo: Unified Adapter and Prompt Learning for Vision Language Models
Combining adapters and prompts to improve few-shot generalization in vision-language models.
Research direction
Computational Photography
2021–2023
Earlier work focuses on efficient image restoration and physically grounded evaluation. V-DESIRR (ICCV 2021) is a multi-scale network for single-image reflection removal, addressing weak through strong reflections. Measured Albedo in the Wild (MAW, ICCP 2023) introduces measured-albedo data and complementary metrics for evaluating intrinsic image decomposition. This strand sits alongside my broader work on inverse rendering and video quality assessment, including the video enhancement article linked under Miscellaneous.
Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluation
A measured-albedo dataset and evaluation metrics for intrinsic image decomposition.
V-DESIRR: Very Fast Deep Embedded Single Image Reflection Removal
A multi-scale architecture for fast single-image reflection removal.