Prof. Mohamad Sawan Center's Research on Autonomous Driving Edge Perception Published in Top Robotics Journal IEEE Robotics and Automation Letters (RA-L)

In Advanced Driver-Assistance Systems (ADAS), conventional computer vision algorithms rely on frame-based RGB cameras, which suffer from high latency in high-speed or sudden scenarios due to fixed frame rates. Furthermore, their performance degrades under demanding conditions like overexposure, inadequate illumination, and fast motion.

To address these issues, bio-inspired event cameras (e.g., DVS) offer microsecond-level temporal resolution and a high dynamic range (up to 120 dB) by asynchronously recording brightness changes. However, the sparse and asynchronous nature of event data makes it challenging to achieve accuracy comparable to frame-based algorithms.

Thus, a key challenge for brain-inspired algorithms is to effectively fuse the rich spatial context of RGB frames with the high temporal resolution of event streams, enabling efficient, low-latency object detection.

To tackle the challenge of fusing spatiotemporal information from RGB cameras and event cameras, the CenBRAIN Neurotech Center of Excellence at Westlake University has developed a novel hybrid neural network architecture known as Event-Fused Hybrid (EFH). The work has been published in IEEE Robotics and Automation Letters (RA-L), a top journal in robotics and automation. Targeting edge perception scenarios including high-level autonomous driving and industrial high-speed robotic vision, the study shows strong potential for real-world industrial adoption and provides valuable insights for future engineering applications.

Fig.1 Hybrid ANN-SNN architecture that processes high-rate event-stream data using SNNs and outputs object detection results at each time-step.

The paper's first author is Chengjun Zhang, an engineer at the Westlake Institute for Optoelectronics. The co-corresponding authors are Dr. Jie Yang, a Research Professor at Westlake University, and Chair Professor Mohamad Sawan. The research was supported by the Science and Technology Innovation 2030 – "Brain Science and Brain-Inspired Intelligence" Major Project.

Abstract

This study proposes an Event-Fused Hybrid (EFH) architecture for low-latency object detection in automotive vision.

EFH combines Artificial Neural Network (ANN) for static feature extraction from RGB frames with Spiking Neural Network (SNN) that dynamically update these features using event streams. This approach enables high-efficiency, high-frame-rate object detection by effectively fusing asynchronous event data, achieving a theoretical throughput of up to 200 FPS.

Experimental results on benchmarks such as DSEC-Detection and PKU-DAVIS-SOD demonstrate that EFH achieves state-of-the-art performance and exhibits strong robustness in motion blur and low-light scenarios. Furthermore, the system was deployed on a vehicle platform, achieving real-time detection and perform significant advantage in edge scenarios, while the SNN branch significantly reduced power consumption during event processing.

Fig.2 Robot platform and its detection performance in edge scenarios.

Research Highlight
a) Brain-inspired Hybrid Architecture:

The model innovatively integrates ANNs for static imagery with SNNs for dynamic asynchronous event streams. SNNs natively process spike-based inputs, preserving the original spatiotemporal structure without handcrafted representations.

b) CLDKC Fusion Module:

A Cross Large-Dynamic-Kernel Convolution (CLDKC) module is introduced. It utilizes dynamic kernels with large receptive fields to adaptively fuse static image context with dynamic event information, enhancing discriminative power for object localization.

c) Low Latency & Low Power:

By performing continuous inter-frame inference using SNNs, the framework mitigates the "blind intervals" of frame-based sensors. Furthermore, leveraging the spike sparsity of SNNs significantly reduces computational energy consumption.

Paper Information
C. Zhang, Y. Zhang, J. Yu, J. Yang and M. Sawan. Event-Fused Hybrid ANN-SNN Architecture for Low-Latency Object Detection in Automotive Vision. IEEE Robotics and Automation Letters, vol. 11, no. 3, pp. 3622-3628, March 2026.

Click the link to access the full paper:
https://doi.org/10.1109/LRA.2026.3662637