KI-ASIC

AI processor architectures for radar modules in autonomous vehicles

The KI-ASIC project is a public research project funded by the Federal Ministry of Education and Research (BMBF) of Germany (official website, in German). Based on recent advances in brain inspired computational hardware, namely neuromorphic computing [1], the purpose of the project is to study, design, and implement neuromorphic solutions for automotive radar processing. The specific role of TUM in the project is to explore new techniques that improve state-of-the-art neuromorphic computing algorithms using so called spiking neural networks (SNNs). These networks are said to be the third generation of artificial neural networks (ANNs) [2]. They operate by propagating only binary, asynchronous spikes of information between neurons - analogous to the mammalian brain and in contrast to "traditional" ANNs, which are based on matrix operations. This sparse communication scheme potentially leads to a decrease in computation time as well as the required power for the system. The combination of these SNNs with neuromorphic hardware is therefore a promising candidate for sensor data processing in automotive applications.

Wihin the project, we collaborate with several partners from industry (Infineon AG, BMW) as well as other universities (TUD, OTH Amberg-Weiden) in order to achieve multiple, overall goals:

Dataset creation using vision, lidar and radar data (OTH)
Development of a new neuromorphic hardware board - SpiNNaker 2 (TUD)
Development and integration of a high resolution radar sensor (IFAG)
Development, evaluation and integration of SNNs on neuromorphic hardware for radar data processing (TUM, IFAG, TUD)

The focus of our research can be divided in the following areas:

Design and simulation of novel spiking neural networks (SNNs), including coding techniques, neuron models, architectures, or learning approaches
Implementation of SNNs in neuromorphic hardware, e.g., Intel’s Loihi [3], or the Human Brain Project SpiNNaker board [4]
Application of state-of-the-art neuromorphic computing algorithms on real environments
Understanding of state-of-the-art sensor signal processing and creating solid working frameworks that allow to do quantitative comparisons of different approaches
Devise mathematical models that help explain better how the brain works

Research

The overall goal for us in this project is to replace each step of the standard radar processing pipeline for automated driving with a neuromorphic algorithm. The approach that we took for achieving this can be devided into two parts, the sensor data processing of the raw radar data and the object data processing. The latter describes the process of detecting, identifying and tracking objects in pre-processed data, received after the sensor data processing.

Sensor Data Processing

By sensor data processing we understand the early stages of the processing pipeline. This block operates on the raw data from the sensor, and its goal is to prepare the data for higher level algorithms by encoding, filtering noise and artifacts, discarding sections of data that typically do not carry information, and transform the data so it can be more easily interpreted by humans or other algorithms.

In our research, we have designed SNNs that solve the most relevant sensor data processing stages in a radar pipeline:

Encoding raw data to spike trains, avoiding the usage of computationally expensive analogue to digital (ADC) converters
Transforming data to the frequency spectrum, as frequency components carry most information in radar signals [A]
Distinguishing objects from noise in the frequency spectrum, in an operation that is typically called object detection. Our proposed SNNs offer an alternative to the traditional constant false-alarm rate (CFAR) algorithms [B]

We refer interested readers to a paper with the rest of the KI-ASIC partners covering the main topics of automotive radar processing with SNNs [C]. In the animated figure below, there is an example of the data that has been used in the project and the output of the conversion to the frequency spectrum and the object detection.

Object Data Processing

The object data processing includes all algorithms that process the data retrieved from the sensor data processing. Typically, these algorithms are directly targeted towards a certain application, such as object recognition, classification or tracking.

Within the project our focus was mainly on unsupervised learning algorithms:

Clustering of range-doppler/range-angle maps using spiking RBF neurons [5, 6] with spike timing dependent plasticity (STDP)
Single/dual object tracking in range-doppler/range-angle maps with pre-trained/defined dynamic neural fields (DNFs) / continuous attractor networks (CANs)

Moreover, we have also applied the neuron model that we introduced in [A] for the conversion of convolutional deep neural networks into SNNs [D].

The video below demonstrates the object tracking using a DNF. The output of the network is visualized together with the annotated input in the top right image. The DNF consists of a 2-dimensional sheet of neurons connected with a mexican-hat topology. This results in a single acticity bump as output of the network, located wherever the maximum input is, i.e. the object in the scene. While this is very well suited for single object tracking and it does work reasonably well with two objects, the approach is not really suited for multi-object tracking. Nevertheless, we were able to track objects with this technique while maintaining a mean euclidean error of the tracking position around 3.22 m.

Tracking of a car in the Carrada Radar Dataset [7] using a continuous attractor network of spiking neurons.

Contributions

[A] López-Randulfe, J., Reeb, N., Karimi, N., Liu, C., Gonzalez, H. A., Dietrich, R., ... & Knoll, A. (2022). Time-coded spiking fourier transform in neuromorphic hardware. IEEE Transactions on Computers, 71(11), 2792-2802.

[B] López-Randulfe, J., Duswald, T., Bing, Z., & Knoll, A. (2021). Spiking neural network for fourier transform and object detection for automotive radar. Frontiers in Neurorobotics, 15, 688344.

[C] Vogginger, B., Kreutz, F., López-Randulfe, J., Liu, C., Dietrich, R., Gonzalez, H. A., ... & Mayr, C. (2022). Automotive radar processing with spiking neural networks: Concepts and challenges. Frontiers in Neuroscience, 414.

[D] López-Randulfe, J., Reeb, N., & Knoll, A. (2022). Conversion of ConvNets to Spiking Neural Networks With Less Than One Spike per Neuron. In 2022 Conference on Cognitive Computational Neuroscience (pp. 553-555).

References

[1] B. Rajendran, A. Sebastian, M. Schmuker, N. Srinivasa, and E. Eleftheriou, “Low-Power Neuromorphic Hardware for Signal Processing Applications,” arXiv:1901.03690 [cs], Jan. 2019, Accessed: Jul. 23, 2019. [Online]. Available: http://arxiv.org/abs/1901.03690

[2] W. Maass, “Networks of spiking neurons: The third generation of neural network models,” Neural Networks, vol. 10, no. 9, pp. 1659–1671, Dec. 1997, doi: 10.1016/S0893-6080(97)00011-7.\

[3] M. Davies et al., “Loihi: A Neuromorphic Manycore Processor with On-Chip Learning,” IEEE Micro, vol. 38, no. 1, pp. 82–99, Jan. 2018, doi: 10.1109/MM.2018.112130359.

[4] S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, “The SpiNNaker Project,” Proceedings of the IEEE, vol. 102, no. 5, pp. 652–665, May 2014, doi: 10.1109/JPROC.2014.2304638.

[5] J. J. Hopfield, “Pattern recognition computation using action potential timing for stimulus representation,” Nature, vol. 376, no. 6535, Art. no. 6535, Jul. 1995, doi: 10.1038/376033a0.

[6] T. Natschläger and B. Ruf, “Spatial and temporal pattern analysis via spiking neurons,” Network: Computation in Neural Systems, vol. 9, no. 3, pp. 319–332, Jan. 1998, doi: 10.1088/0954-898X_9_3_003.

[7] A. Ouaknine, A. Newson, J. Rebut, F. Tupin, and P. Pérez, “CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations,” arXiv:2005.01456 [cs], Feb. 2021, Accessed: Mar. 08, 2021. [Online]. Available: http://arxiv.org/abs/2005.01456