Why Neither Modality Alone Is Enough

Visual perception and force/torque sensing each fail at precisely the moment the other succeeds — and in unstructured workcells, those failure moments are unavoidable. Vision is highly sensitive to pose and posture changes between assembled objects but cannot reliably reflect contact state, especially when parts occlude each other during insertion. Force/torque sensing captures contact richness but is insensitive to macroscopic positional and orientation changes. This complementary failure structure, documented across more than 50 patents and research papers spanning 2013 to 2025, is the foundational motivation for multi-modal fusion in robotic assembly.

The dataset — with a pronounced concentration of publications between 2020 and 2023 — reflects an industry-wide recognition that fixture-free, unstructured assembly is the next frontier for autonomous manufacturing. Parts move freely, surface conditions vary, and no two cycles present identical contact geometry. In these conditions, a robot relying solely on RGB-D cameras will lose track of part state the moment a gripper finger occludes the target; a robot relying solely on wrist force/torque will be unable to identify which part it has contacted or plan a corrective approach trajectory. According to WIPO trend data, patent filings in AI-guided robotic manipulation have grown substantially since 2018, with sensor fusion methods representing a growing share of claims.

The practical consequence of this gap was quantified empirically by benchmarking on the NIST Assembly Task Boards (2021), which showed that neither hybrid force/motion control nor 2D/3D pattern matching in isolation achieves reliably high success rates across the full task range. The bottleneck problems differ between vision-based localization and force-based contact management, and an end-to-end integrated solution combining both modalities outperformed the separated baselines. This result provides the comparative quantitative evidence that motivates multi-modal system investment for unstructured workcell deployment.

Sensor Fusion Architectures: From Bayesian Estimators to Tensor Networks

The dominant sensor fusion architectures share a common structural principle: sensing modalities are decoupled at the hardware level and integrated at the representation level, so that each stream contributes its specific strength without being degraded by the other’s noise characteristics. Three architectures have emerged as the most technically mature across the patent and research literature.

The Bayesian unification approach, formalized by researchers at TU Munich (2021), presents a single Bayesian framework that continuously tracks part poses using both visual observations and intrinsic tactile sensing. The framework enables a robot to maintain accurate state estimates even when occlusion blocks the vision system, allowing object-centric assembly skills guided by these estimated poses to proceed robustly through phases where either modality alone would produce state estimation collapse. This approach is particularly significant in unstructured settings where parts move freely and are not constrained by fixtures.

The tensor fusion network architecture is explicitly commercialised in a 2025 patent from Harbin Institute of Technology, Shenzhen. This system provisions a plurality of neural network models — including a reinforcement learning network and a tensor fusion network — trained simultaneously on visual data, tactile sensor data, robot motion feedback, and torque feedback. Tactile signals are used to indirectly infer multi-dimensional external forces, and the resulting fused representation vector generates robot motion instructions that flexibly adjust insertion force to complete assembly. The architecture decouples sensing modalities at hardware level while integrating them at representation level, enabling the system to respond to contact events that pure vision would miss entirely.

A self-supervised deep RL pipeline — where vision governs coarse alignment and force/torque judges contact correctness — was demonstrated by Shenyang University of Technology (2021) for peg-in-hole assembly. The agent first observes the environment through vision to plan alignment actions, then uses force/torque feedback to judge contact correctness, achieving a combined perception pipeline that surpasses either modality alone. Separately, Autodesk Research (2021) established a strong unimodal force-control baseline using a Recurrent Distributed DDPG agent trained on force/torque in task space as the sole observation, enabling robot-agnostic transfer across arm morphologies without re-training. This baseline makes clear that the benefit of adding vision is most pronounced for initial coarse localization and part identification — tasks force sensing cannot perform — while force feedback governs the contact-rich insertion phase.

“The benefit of adding vision is most pronounced for initial coarse localization and part identification — tasks force sensing cannot perform — while force feedback governs the contact-rich insertion phase.”

Learning Methodologies: Primitives, Imitation, and Sim-to-Real Transfer

Three learning methodologies dominate the literature for training multi-modal assembly policies: manipulation primitive decomposition, hierarchical imitation learning that separates trajectory and force streams, and sim-to-real transfer augmented by visual error estimators and domain randomization.

Manipulation Primitives with Contact Semantics

NTU Singapore (2021) demonstrated that decomposing assembly into semantically meaningful manipulation primitives — such as “Move down until contact” or “Slide along x while maintaining contact with the surface” — keeps the reinforcement learning search tree shallow while encoding physical contact semantics. Policies learned entirely in simulation using these primitives achieve direct sim-to-real transfer on tasks including round peg insertion with 0.04 mm clearance without retraining. The contact-awareness embedded in each primitive is precisely what allows sim-to-real transfer: the policy does not need to model the visual appearance of contact, only the force-defined event that terminates or transitions the primitive.

Hierarchical Imitation Learning

AIST Japan (2021) addressed a fundamental limitation of learning from demonstration: while nominal trajectories can be captured via kinesthetic teaching or teleoperation, realistic contact force profiles cannot be reliably obtained from simulation due to the reality gap. The proposed hierarchical imitation learning framework learns trajectory from demonstration while learning the force profile separately through physical interaction, then combines both to generalise across assembly configurations. This separation of concerns is architecturally important: the vision stream governs trajectory generation; the force stream governs contact compliance. The approach is consistent with findings from IEEE research on human-robot skill transfer, which similarly highlights the difficulty of capturing force profiles through observation alone.

Sim-to-Real Transfer and Visual Error Correction

Hunan University (2023) employs an RGB-D camera to capture 6D pose teaching trajectories from human demonstrations, uses these to pre-train an assembly policy in simulation, and then deploys a visual error estimator — derived from the gap between simulated and real robot states — to correct policy outputs during real-world execution. Domain randomization further improves robustness. This combination of visual teaching, sim pre-training, and a learned error-correction model directly targets the success-rate degradation that occurs when simulation-trained policies encounter real-world visual noise.

Meta-reinforcement learning compresses the real-world data requirement further. Siemens (2020) demonstrated that training on a family of simulated insertion tasks and then adapting to real-world tasks in fewer than 20 real trials substantially reduces the data requirement that would otherwise make force-and-vision RL impractical in production settings. This approach is especially relevant for manufacturers deploying across multiple product variants, where per-variant retraining would be prohibitively expensive.

Stanford University (2021) demonstrated a further learning methodology: hierarchical failure recovery through contact interpretation. This probabilistic approach trains differentiable filters that exploit the tactile sensorimotor trace from failed assembly attempts to update belief about part position and type — achieving higher precision in position and type estimation and completing fitting tasks faster than baselines. By treating assembly failures as informative force-tactile signals rather than terminal states, the system effectively closes the loop between detection of failure (via force) and re-localization (updating visual-tactile belief state). This approach, consistent with active perception frameworks studied at institutions such as NIH-funded neuroscience labs researching sensorimotor integration, treats each failed attempt as a data point rather than a reset event.

Engineering Implementations in Unstructured Workcells

Industrial deployment of multi-modal systems reveals a consistent architectural pattern: RGB-D or structured-light vision handles coarse part localization and grasp planning, while force/torque control governs the contact-rich insertion phase. The transition between modality-dominant phases is a critical engineering decision that differentiates high-performing systems.

Robotic Materials Inc. (2020) benchmarked this pattern directly, evaluating peg-in-hole and hole-on-peg assemblies from the 2018 World Robotics Summit challenge across 20 experimental trials per task. The system uses RGB-D sensing for initial part localization and grasp planning, then transitions to spiral-based search and tilting insertion algorithms governed by hand-coded force/torque thresholds to detect critical assembly transitions. This industrial trial directly quantifies how the combination of 3D vision (for coarse pose) and force control (for fine insertion) enables reliable completion of tasks that neither modality could handle alone.

Agile Robots AG’s second contribution — proactive visual-haptic residual RL — addresses target uncertainty in unstructured environments by incorporating both visual observation and torque feedback into the RL policy and introducing a proactive action mechanism to resolve partial observability. The system is validated on RAM module insertion — a highly contact-rich, precision-demanding task — demonstrating that the visual-haptic fusion improves sample efficiency of policy learning, not only final success rate. This distinction is practically important: reduced sample efficiency means more real-world trials, which is costly in production environments.

Autodesk Research (2022) demonstrated CAD-informed adaptive assembly in a two-robot workcell that assembles interlocking 3D designs by combining CAD-derived simulation training with real-time visual inference. The system’s ability to operate without task-specific fixtures exemplifies the unstructured workcell paradigm and aligns with the broader trajectory toward generative-design-to-autonomous-fabrication pipelines that standards bodies such as ISO are beginning to address in their robotic manufacturing standards.

Osaka University’s earlier work (2017) established the multi-modal vision framework that later papers extend. The system addresses the precision problem of 3D visual detection in occlusion-prone unstructured scenes by switching sensing modality depending on task phase — using AR markers in a teaching phase and point clouds with geometric constraints in a robot execution phase — and integrates the result with a graph-model-based assembly planner. This phase-switching strategy foreshadows the more sophisticated learned transition policies in later work.

Key Players and the Shift Toward Learned Adaptive Policies

Several institutions and companies recur as sustained contributors across the 50+ paper and patent dataset, each with a distinct technical focus that collectively maps the field’s intellectual geography.

University of Patras – Laboratory for Manufacturing Systems and Automation is the single most frequent institutional contributor, appearing across papers on knowledge-enabled cell design, AI-based scheduling, mobile dual-arm workers, ROS-based human-robot collaboration frameworks, augmented reality operator guidance, and machine-learning-based assembly monitoring — exemplified by its 2021 work on seamless human-robot collaborative assembly using AI and wearable devices.

Harbin Institute of Technology (Harbin and Shenzhen campuses) produces work spanning visual teaching for task learning (2020), prediction-based human-robot collaboration via ConvLSTM (2022), and the visual-tactile tensor fusion patent (2025) — a trajectory that tracks the field’s evolution from demonstration-based methods toward end-to-end learned fusion architectures.

Agile Robots AG contributes two high-impact results: the pushing-based hybrid position/force skill achieving 100% success rate, and the proactive visual-haptic residual RL for torque-controlled insertion — both grounded in unstructured industrial scenarios. Autodesk Research pursues the design-to-manufacture pipeline, with force-guided robot-agnostic assembly RL and CAD-informed adaptive assembly workcells. KTH Royal Institute of Technology addresses multimodal human-robot collaboration control, particularly the fusion of haptic, gesture, and voice inputs for symbiotic assembly. These institutional profiles are consistent with global R&D patterns tracked by OECD in its science and technology outlook reports on intelligent manufacturing.

The overarching trend from the data is a temporal evolution from hand-coded force thresholds and fixed vision pipelines (2017–2019) toward learned, adaptive multi-modal policies (2020–2025), with increasing emphasis on sim-to-real transfer as the mechanism bridging the gap between training efficiency and deployment reliability in unstructured workcells. The 2021 concentration of landmark results — 100% success rate from Agile Robots AG, direct 0.04 mm clearance sim-to-real transfer from NTU Singapore, failure-recovery through contact interpretation from Stanford, and Bayesian visual-tactile fusion from TU Munich — marks the field’s transition from proof-of-concept to production-relevant capability. This trajectory is consistent with broader AI-in-manufacturing trends documented by WIPO in its 2023 Technology Trends report on AI.

“Treating failed insertion attempts as informative sensorimotor signals — rather than terminal states — and using differentiable Bayesian filters to update part-type and position belief produces faster task completion and higher accuracy than retry-from-scratch strategies.”

For R&D teams investing in unstructured workcell automation, the patent literature points to three non-negotiable design decisions: (1) the fusion architecture must integrate modalities at the representation level, not just sequentially; (2) manipulation primitives or hierarchical decomposition must be used to make sim-to-real transfer tractable; and (3) failure recovery through force-tactile belief updating must be built into the policy architecture from the start. Systems that satisfy all three conditions consistently outperform those that address only one or two. The PatSnap R&D Intelligence platform enables teams to map the patent landscape across all three dimensions simultaneously, identifying white spaces and competitive overlaps before committing to an architecture.