Under Double-Blind Review
Multimodal Interaction Estimation for Contact-Aware Adaptive Impedance Control
Fusing force history and 3D point cloud through a transformer-based classifier to distinguish unknown payloads from human contact, and to drive a variable-impedance controller in real time.
Abstract
With advancements in learning-based policies, robots are increasingly being deployed in human-centric environments, such as manufacturing floors and homes, where safe and precise operation requires compensating for unknown loads while remaining compliant during human interactions. Reliably distinguishing between these interaction types is challenging with a single sensing modality, as force signals cannot separate payloads from human pushes, and vision is susceptible to occlusions. We propose a task-agnostic interaction estimation and adaptive control framework that addresses this challenge by fusing force history and 3D point cloud through a transformer-based classifier. Trained with modality dropout for occlusion resilience and fine-tuned from simulation to hardware, the classifier achieves 91.89% accuracy and maintains above 90% under complete occlusion of one modality. The classifier drives a system identification framework that selectively compensates for unknown payloads and a variable-impedance controller that switches between adaptive and reactive modes, while simultaneously estimating interaction forces for downstream tasks, such as human intent estimation. We validate the framework under external perturbations and unknown loads across single-arm manipulation, deployment with an imitation-learning-based policy, intent-driven target switching, and zero-shot bimanual deployment on two Franka arms lifting a box of unknown mass, demonstrating generalization to closed kinematic chains and unseen morphologies, enabling safe and precise deployment.
Experiments
Validated across single-arm, intent-driven, and bimanual manipulation.
Adaptive Impedance Control
To examine the controller's accuracy in the continuous adaptation, we deploy the interaction classification and adaptive control framework on the Franka arm and continuously apply loads and perturbations. The framework distinguishes load- and interaction-induced forces online, adjusting impedance parameters and gravity compensation in real-time. As shown in the video, this enables compliant behavior under human contact while maintaining precise trajectory tracking in the reactive mode.
Intent Estimation
To evaluate interaction force estimation, we use the estimate as the input to a Hidden Markov Model-inspired target belief filter, which estimates the robot's next target by evaluating the alignment of the interaction force with the end-effector-to-target distance vector. The adaptive controller provided accurate interaction force estimation, enabling reliable target inference and a collaborative human-robot-human transportation task, as shown in the video.
Bimanual and Contact-Rich Manipulation
We deploy adaptive controller in contact-rich tasks, such as pushing and bimanual manipulation (unseen in training), simultaneously lift a box of unknown mass, thereby testing adaptation within a closed kinematic chain, subject to external perturbation. The same RME framework and interaction classifier as in the single-arm experiment are used (both robots use the more conservative of the two individual estimates); the classifier is fine-tuned only on real-robot data for better sim-to-real transfer on unseen setup. Subtracting the desired contact force from the external force lets the classifier distinguish desired contact from perturbations, enabling compliant yet precise contact-rich tasks.
Imitation Learning Policy
To evaluate our framework with an imitation-learning policy, we deploy the EMP policy (Li et al.) for a pouring task involving an unknown 0.85 kg payload, which the policy was not trained to handle, and compare fixed-gain controllers against our adaptive impedance controller (with the classifier fine-tuned only on real-robot data to improve sim-to-real transfer to this unseen setup). The soft controller yields to the applied payload and fails to complete the task in all three trials, diverging from the trajectory and colliding with the table. The stiff controller improves tracking; however, it fails to pour the payload in two trials due to collision with the pot and an inability to complete the task, as it cannot compensate for the unknown dynamics. The adaptive controller, which adjusts impedance and gravity compensation online, most closely follows the EMP trajectory and succeeds in all three trials, allowing the controller to safely and precisely execute the learned policy with unknown loads while maintaining compliance to human perturbations.
Perturbations Detection
Deployed on the Franka arm tracking equilibrium point with the controller, the classifier achieves high accuracy in detecting continuous perturbations when both physical contact and loads are present. The joint latent representation of force and point cloud data enables the model to confidently distinguish interactions in complex scenarios, even when the force profiles resemble the applied load (a perturbation along the z-axis). In gray, we label time intervals during which a perturbation was applied to the robot, and in red, we mark time intervals when the NN predicted a compliant mode.
Implementation Details and Proofs
Network Architecture
See DetailsThe interaction classifier fuses a force-history encoder and a point-cloud encoder through a shared transformer with a learned prediction token. It is trained with modality dropout for occlusion resilience, first in simulation and then fine-tuned on real-robot data.
Intent HMM
See DetailsWe model target inference as a Hidden Markov Model over candidate targets, with transition and emission probabilities derived from the alignment between the estimated interaction force and the end-effector-to-target direction.
Passivity Proof
See DetailsWe establish passivity of the switching variable-impedance controller across adaptive and reactive modes, bounding the net energy injected at the interaction port and guaranteeing stable coupling with an unmodeled human or environment.