A new AI system called a semantic decoder can translate a person’s brain activity — while listening to a story or silently imagining telling a story — into a continuous stream of text.
The system, developed by a team at The University of Texas at Austin, might help people who are mentally conscious yet unable to physically speak, such as those debilitated by strokes, to communicate intelligibly again.
The research, published in the journal Nature Neuroscience, was led by Jerry Tang, a doctoral student in computer science, and Alex Huth, an assistant professor of neuroscience and computer science at UT Austin.
“For a non-invasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences.
“We’re getting the model to decode continuous language for extended periods of time with complicated ideas.”
Unlike other language decoding systems in development, the system does not require subjects to have surgical implants.
Participants also do not need to only use words from a prescribed list.
Brain activity is measured using an fMRI scanner after extensive training of the decoder, with the individual listening to hours of podcasts in the scanner.
Later, provided that the participant is open to having their thoughts decoded, their listening to a new story or imagining telling a story enables the machine to generate corresponding text from brain activity alone.
The result is not a word-for-word transcript. Instead, researchers have designed it to capture the gist of what is being said or thought, albeit imperfectly.
Around half the time, when the decoder has been trained to monitor a participant’s brain activity, the machine produces text that closely (and sometimes precisely) matches the intended meanings of the original words.
The researchers addressed questions about potential misuse of the technology in a paper.
The researchers explained how decoding worked only with cooperative participants who had participated willingly in training the decoder.
Results for individuals on whom the decoder had not been trained were unintelligible, and if participants on whom the decoder had been trained later put up resistance — by thinking other thoughts, for example — results were similarly unusable.
“We take very seriously the concerns that it could be used for bad purposes and have worked to avoid that.
“We want to make sure people only use these types of technologies when they want to and that it helps them.”
In addition to having participants listen or think about stories, the researchers also asked subjects to watch four short, silent videos while in the scanner. The semantic decoder was able to use their brain activity to accurately describe certain events from the videos.
The system currently is not practical for use outside of the lab because of its reliance on the time need on an fMRI machine.
But the researchers believe that this work could transfer to other, more portable brain-imaging systems, such as functional near-infrared spectroscopy (fNIRS).
“fNIRS measures where there’s more or less blood flow in the brain at different points in time, which, it turns out, is exactly the same kind of signal that fMRI is measuring.
So, our exact kind of approach should translate to fNIRS,” however, he noted, the resolution with fNIRS would be lower.
Image: Nolan Zunk/University of Texas at Austin
Digital twins set to advance stroke care
Can AI help keep the human experience in social care?
mRNA discovery paves the way to safer vaccines
Eye scans provide crucial insights into kidney health
Key tips for accelerating the health security compliance journey
Nordic Consulting strengthens team in Ireland
Healthcare innovators and leaders honoured at Imprivata HealthCon user group and awards ceremony
Wearable communication system may reduce digital health divide
Molecule trains the immune system to prevent cancer
Diabetes patients urged to use fitness games with caution
- UK3 weeks ago
National first for openEHR portal and e-referrals solution
- Medtech4 days ago
eSight: “The technology has the potential to change someone’s life”
- News4 weeks ago
How could technology better support patient experience in the NHS?
- Entrepreneurship2 weeks ago
‘I was producing addicts every week’ – the doctor-turned-tech leader taking on the opioid crisis