As a global leader in the commercialization of autonomous driving, Autowise.ai has deployed multiple autonomous driving products across over 30 cities worldwide. To meet the varied demands of multi-region, multi-scenario, and multi-product operations, Autowise has integrated cutting-edge AI technologies and algorithms. Their new generation 1vN autonomous driving remote operations system incorporates advanced multimodal large-model technology, acting as an “AI Safety Officer” to bolster safety for autonomous vehicles with robust scene understanding and accurate decision-making capabilities.
AI Safety Officer Uses Multimodal Large Model to Analyze Real-Time Image Data and Identify Potential Hazards
The multimodal large model represents a significant advancement in AI, integrating various data forms—text, images, audio, and video—into a unified framework that enables cross-modal understanding and generation. This technology supports diverse tasks, such as image description, visual Q&A, and video generation. In autonomous driving, the AI Safety Officer processes and analyzes camera images in real-time, identifying hazardous scenes and abnormal situations that the vehicle may encounter, thus ensuring safer operations.
Tackling Long-Tail Scenarios Beyond Algorithmic Coverage
In addition to interacting with conventional road participants, autonomous vehicles must handle complex, rare “long-tail” scenarios—situations that occur infrequently but pose significant challenges to autonomous systems. Autowise’s autonomous sweepers, for example, often operate close to the curb, where they encounter unique challenges such as irregular potholes, temporary construction zones, or unexpected traffic incidents.
These long-tail scenarios are critical because they represent rare yet potentially dangerous situations that an autonomous driving system must navigate. Human drivers rely on intuition and experience to handle these edge cases. However, autonomous systems need vast data and complex algorithms to simulate this decision-making process. Given the complexity of real-world conditions, algorithms alone cannot cover all scenarios. Here, the AI Safety Officer steps in, leveraging near-human knowledge and experience to ensure safe navigation.
Real-World Applications of the AI Safety Officer: Scene Interpretation and Advisory Responses
Autowise has developed an abnormal dataset for decision-making and planning based on historical takeover data and closed-loop simulation data, leveraging model annotation and human alignment. Enriched with real-world operational data, the AI Safety Officer’s multimodal large model will offer increasingly accurate scene interpretation and driving recommendations.
Below are some examples illustrating its capabilities:
Sudden Traffic Accidents
While an autonomous sweeper operates on a side lane, it encounters a sudden traffic accident—say for example a delivery rider skids on a wet road. The system flags a “Traffic Accident” status code and alerts for possible remote intervention. Fortunately, the rider is unharmed, and the system advises slowing down and proceeding cautiously.
Navigating Temporary Road Closuresand Construction Zones
The sweeper encounters both construction work on the side lane and overhead work on the main road, with traffic police directing vehicles. The system flags “Traffic Police Control” and “Construction Area” status codes and prompts for potential remote intervention. With remote assistance, the sweeper safely reroutes through an adjacent lane.
Emergency Vehicle Interaction
While performing curbside sweeping, the vehicle spots an ambulance with flashing lights parked roadside, indicating a nearby emergency. The system raises an “Emergency Vehicle” status code, prompting remote intervention and, when safe, advises the sweeper to slow down and yield.
In a narrow lane, due to a parked vehicle, the sweeper detects children and crouched pedestrians nearby, raising a “High-Risk Pedestrians” status code. It advises reduced speed and alerts for possible remote intervention. Using high-precision perception, the sweeper makes a slight avoidance maneuver and proceeds slowly.
AI Provides Safety Backbone, While Human Operators Offer Additional Support for Emergency Situations
In specific emergency or extreme scenarios, the AI Safety Officer prompts human remote safety operators for intervention. These operators receive real-time video feeds (front and 360-degree views) and sensor data (high-precision maps, obstacle detection, path planning) to assess the vehicle’s environment and provide support to help the vehicle safely navigate. Autowise’s autonomous vehicles currently support various remote-assist driving options—including reference-line, steering, app-based, and cockpit controls—providing an extra layer of safety in emergencies.
The AI Safety Officer, powered by the cloud-based large model, significantly enhances the autonomous vehicle’s ability to recognize and understand long-tail scenarios. This improvement also increases the efficiency of human remote operators, laying a strong technical foundation for the ongoing expansion of 1vN-scale remote operations and broadening Autowise’s global commercial reach.
Looking ahead, Autowise will continue to adopt and integrate leading-edge AI technologies and algorithms, innovating to improve the safety and reliability of autonomous driving technology. These advancements will form a solid foundation for the rapid commercialization of autonomous driving, contributing to the global transformation of autonomous driving and AI industries.
