In-sensor neural network for real-time KWS by image processing

Paola Vitolo; Pio Esposito; Danilo Pau; Rosalba Liguori; Luigi Di Benedetto; Gian Domenico Licciardo

doi:10.1117/12.2665545

7 June 2023 In-sensor neural network for real-time KWS by image processing

Paola Vitolo, Pio Esposito, Danilo Pau, Rosalba Liguori, Luigi Di Benedetto, Gian Domenico Licciardo

Proceedings Volume 12571, Real-time Processing of Image, Depth and Video Information 2023; 125710F (2023) https://doi.org/10.1117/12.2665545
Event: SPIE Optics + Optoelectronics, 2023, Prague, Czech Republic

Abstract

KeyWord Spotting (KWS), i.e. the capability to identify vocal commands as they are pronounced, is becoming one of the most important features of Human-Machine Interface (HMI), also thanks to the pervasive diffusion of high-performance MEMS audio sensors with very reduced dimensions. In-Sensor Computing (ISC) appears the most viable solution to get the maximum advantage of KWS, since the dimensions of MEMS microphones remain reduced and minimally invasive. ISC, indeed, represents the extreme evolution of the edge computing paradigm, where the processing circuits are moved close to the audio sensor, integrated into its auxiliary circuitry or in the same package. However, ISC introduces severe area and power constraints and must trade off with processing speed to meet real-time operations naturally required by KWS. In this work, we want to show a neural network-based KWS suitable for ISC contexts, when audio sensor data are converted into MEL spectrogram images and a Depthwise Separable Convolutional Neural Network (DSCNN) with feature extraction capabilities is designed. To show the advantages of the above approach, the DSCNN is compared with an alternative Fully Connected Neural Network (FCNN), operating on audio signals not converted into images. The considered models have been profiled on a microcontroller and implemented on an FPGA. Their performances are compared in terms of classification accuracy and HW resources. Comparisons show that the FCNN is very far from meeting the ISC real-time processing requirements, showing a number of parameters and a frame latency respectively of 3 and 1 orders of magnitude higher than required by the DSCNN alternative when mapped to a Xilinx Zynq Ultrascale+ MPSoC.

Conference Presentation

Citation Download Citation

Paola Vitolo, Pio Esposito, Danilo Pau, Rosalba Liguori, Luigi Di Benedetto, and Gian Domenico Licciardo "In-sensor neural network for real-time KWS by image processing", Proc. SPIE 12571, Real-time Processing of Image, Depth and Video Information 2023, 125710F (7 June 2023); https://doi.org/10.1117/12.2665545

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available