CenBRAIN Neurotech, NLP Lab and SAHZU Make Progress in Chinese Language Decoding Brain-Computer Interface Technology

Brain-computer interface (BCI) technology is recognized as the main battlefield for the future integration of life sciences and information technology, with significant social value and strategic importance. On November 6, 2023, the team from CenBRAIN at Westlake University, led by Prof. Sawan, together with Prof. Yue Zhang's team from the Natural Language Processing Laboratory, and Prof. Junming Zhu's team from the Second Affiliated Hospital of Zhejiang University, jointly announced their latest research discovery: "A high-performance brain-sentence communication system designed for logosyllabic language." This research pioneers the full-spectrum Chinese language decoding for BCIs and advancing the field of BCI filling a gap in Chinese language decoding technologies at the international level.

BCI technology creates a communication pathway between the human or animal brain and external devices, essentially serving as a new channel for information exchange, allowing direct communication with the outside world by bypassing the traditional muscular and peripheral nervous pathways, and thus can substitute functions such as human motion and speech. In August this year, two back-to-back articles in "Nature" demonstrated the powerful capabilities of BCIs in language restoration. However, existing language BCIs are predominantly developed for alphabetic languages like English, and research on BCI systems for non-alphabetic languages like Chinese is still lacking. This study utilized Stereo-Electroencephalography (SEEG) to collect neural activity signals corresponding to the pronunciation process of all Mandarin characters, combined with deep learning algorithms and language models, to decode the full spectrum of Mandarin pronunciation for the first time, establishing the world's first Chinese BCI system that covers all Mandarin characters, achieving end-to-end output from brain activity to complete Mandarin sentences.

Chinese, as a logographic and syllabic language with over 50,000 characters, is significantly different from English, which is composed of 26 letters, presenting a huge challenge to existing language BCI systems. To address this problem, over the past three years, Westlake University's research team has thoroughly analyzed the phonetic elements and characteristics of Mandarin. Starting with the three elements of Mandarin pronunciation—initials, tones, and finals—and integrating the characteristics of the pinyin input system, the team designed a new BCI system for Chinese. By creating a phonetic library covering all 407 Mandarin syllables and the particular features of Mandarin pronunciation and concurrently collecting EEG signals, the team built a database of over 100 hours of Mandarin speech-SEEG. Through AI model training, the system constructed predictive models for the three elements of Mandarin pronunciation and ultimately integrated all predicted elements through a language model, using semantic information to generate the most probable complete Mandarin sentences.

After a systematic performance evaluation, it was found that the brain-computer interface system demonstrated high-performance decoding capabilities in simulating daily Mandarin Chinese environments. Following over 100 randomly selected complex communication scenarios (sentences length ranging from 2 to 15 characters), the median character error rate among all participants was only 29% in average, with some participants achieving a 100% accuracy rate for sentences decoded through ssEEG signals. This efficient decoding performance is attributed to the excellent performance of three independent syllable element decoders and the perfect integration with an intelligent language model. Notably, in classifying 21 initials, the initial decoder's accuracy exceeded 40% (over three times the baseline), with the Top 3 accuracy rate nearly reaching 100%. The tone decoder, which distinguishes four tones, also achieved an accuracy of 50% (over twice the baseline). Besides the significant contributions of the three independent syllable element decoders, the powerful auto-correction and context connection capabilities of the intelligent language model further enhanced the overall performance of the language brain-computer interface system.

This research provides a new perspective for BCI research in logosyllabic languages like Chinese and significantly improves BCI performance through a powerful language model, offering a new direction for future logosyllabic language neuroprosthesis research. This achievement also indicates that soon patients affected by neurological diseases will be able to control computers with their thoughts to generate Chinese sentences, regaining the ability to communicate!