Bridging Perception and Action in Robotics
Multimodal models of large language (MLLMS) are promising to allow machines, such as robotic arms and leg robots, to perceive their environment, interpret scenarios and take significant measures. The integration of such intelligence into physical systems advances the domain of robotics, pushing it towards autonomous machines which do not see and do not describe, but hover and move in their environment according to contextual understanding.
Despite the growing power of the MLLM, a persistent problem is their inability to combine vision, reasoning and physical interaction in a single coherent system. As a rule, the models formed to understand the images or the text fail when they are asked to control robots in real spaces. The central problem is that understanding a scene is fundamentally different from acting. Multimodal understanding focuses on perception and analysis, while physical control must be precise and in real time as a function of this perception. This disconnection creates bottlenecks when they try to build agents who must simultaneously observe, reason and act in various environments.
Limits of previous VLA models
The previous tools designed for robot control are strongly based on vision-action models (VLA). These models train on large robotic data sets to convert visual observations into control signals. While certain solutions try to preserve the reasoning capacity of MLLM by translating orders into textual actions, they are faced with the difficulty of maintaining precision and adaptability during control tasks. For example, VLAs often degrade in performance when applied to various robotic operations or on a long horizon. In addition, due to the gap between image -based understanding and movement control, these tools generally do not manage to generalize in different environments or types of robots.
Presentation of Vebrain: a unified multimodal framework
Researchers from Shanghai AI Laboratory, Tsinghua University and Research on Sensitive Time introduced a unified framework called Visual Embared Brain (Vebrain) in collaboration with several other institutes. Vebrain reformulates the control of robots as textual tasks in a 2D visual space, lining up more closely with the operation of MLLMS. The framework incorporates multimodal understanding, space reasoning and robotic control in a single structure. A specially designed robotic adapter treats the release of the MLLM in executable movement policies, allowing a unique model to manage perception, reasoning and control. Vebrain is also supported by a set of high-quality instructions data called Vebrain-600K, which combines more than 600,000 multimodal tasks samples, including robot movement and reasoning stages.
Technical components: architecture and robotic adapter
To fulfill its functions, Vebrain uses an architecture based on QWEN2.5-VL, increased by components which allow real control. The robotic adapter contains four key modules. The points tracker updates 2D key points as the robot view changes, guaranteeing precise targeting. The motion controller transforms 2D key points into 3D movements by combining image data with depth cards. The skills executor cards have predicted actions, such as “turn” or “enter”, for pre-formulated robotic skills. Finally, the dynamic control module monitors failure or anomalies, putting control to the MLLM when needed. These modules form a closed loop system which makes decisions, acts and self-corrects, allowing robots to operate effectively in various situations.
Performance assessment on multimodal and robotic references
Vebrain was evaluated on 13 multimodal benchmarks and 5 space benchmarks. On MMVET, it reached an improvement of 5.6% compared to QWEN2.5-VL. He obtained a score of 101.5 on the cider metric for Scanqa and scored 83.7 on Mmbench. On the VSI reference, it was on average 39.9, outperforming QWEN2.5-VL 35.9. In robotic assessments, Vebrain showed 86.4% success on seven -legged robotic tasks, considerably exceeding models like VLA and π0, which marked 32.1% and 31.4%, respectively. On robotic arm tasks, he reached a success rate of 74.3%, overpasted from others up to 80%. These results show Vebrain's ability to manage control challenges on a long and spatially complex horizon with high reliability.
Conclusion
The research presents a convincing direction for the embodied AI. The researchers managed to redefine the control of robots as a linguistic task, allowing high level reasoning and low level action to coexist. The method fills the gap between the understanding of the image and the execution of the robot in a way that is both functional and evolving. With a robust design and solid performance, Vebrain signals a change to more unified intelligent robotics systems capable of operating independently through various tasks and environments.
Discover the Paper And GitHub page. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter And don't forget to join our 99K + ML Subreddit and subscribe to Our newsletter.
Nikhil is an intern consultant at Marktechpost. It pursues a double degree integrated into materials at the Indian Kharagpur Institute of Technology. Nikhil is an IA / ML enthusiast who is still looking for applications in fields like biomaterials and biomedical sciences. With strong experience in material science, he explores new progress and creates opportunities to contribute.
