Great Facts: Light decodes what a person sees from brain signals

A man looks in a mirror with another covering his face

Researchers have used light to decode brain signals and identify what image a person sees.

Some people are trapped within their own minds, able to think and feel but unable to express themselves because brain injury or disease has damaged their lines of communication with the outside world.

As a step toward helping people in such situations communicate, researchers have demonstrated that they can use light to detect what is going on inside someone’s head.

As reported in the journal NeuroImage, the researchers use LED light beamed from the outside of the head inward to detect activity in the area of the brain responsible for visual processing, and then decode brain signals to determine what the person sees.

Accomplishing this feat required the development of neuroimaging tools and analysis techniques that move the field a step closer to solving the much more complex problem of decoding language.

The study demonstrates that high-density diffuse optical tomography (HD-DOT)—a noninvasive, wearable, light-based brain imaging technology—is sensitive and precise enough to be potentially useful in applications such as augmented communication that are not well suited to other imaging methods.

“MRI could be used for decoding, but it requires a scanner, and you can’t expect someone to go lie in a scanner every time they want to communicate,” says senior author Joseph P. Culver, professor of radiology at Washington University’s Mallinckrodt Institute of Radiology.

“With this optical method, users would be able to sit in a chair, put on a cap, and potentially use this technology to communicate with people. We’re not quite there yet, but we’re making progress. What we’ve shown in this paper is that, using optical tomography, we can decode some brain signals with an accuracy above 90%, which is very promising.”

Decoding brain signals

When the neuronal activity increases in any region of the brain, oxygenated blood rushes in to fuel the activity. HD-DOT uses light to detect the rush of blood. Participants wear a cap fitted with dozens of fibers that relay light from tiny LEDs to the head. After the light is transmitted through the head, detectors capture dynamic changes in the colors of the brain tissue as a result of changes in blood flow.

Culver, first author and graduate student Kalyan Tripathy, and colleagues set out to evaluate the potential of HD-DOT for decoding brain signals. They started with the visual system because it is one of the best understood brain functions.

Neuroscientists long ago worked out a detailed map of the visual part of the brain by showing participants flashing checkerboard patterns on a screen and identifying the 3D units, known as voxels, in the brain that became active in response to each pattern. Decoding is the attempt to reverse the process: Detecting active voxels and then deducing which checkerboard pattern triggered that pattern of brain activity.

“We know what the participant is seeing, so we can verify how well our decoding matches up to reality,” says Culver, also a professor of physics and of electrical and systems engineering and biomedical engineering at the McKelvey School of Engineering. “By going to something that was well validated, we could optimize the experimental design, push harder on the statistics of the decoding, and obtain performance that is really very high.”

From checkerboard to communication

The researchers started simple. They recruited five participants for multiple five- to 10-minute runs in which the participants were shown a checkerboard pattern on either the left or the right side of the visual field for a few seconds at a time, interspersed with breaks during which no image was shown.

Using one run as the template, the researchers analyzed the data from another run to determine when the checkerboard was on which side of the screen. They repeated this analysis using different runs as the template and the test until they had analyzed all possible pairings.

The researchers were able to identify the correct position of the checkerboard—left, right, or not visible at all—with 75% to 98% accuracy. While decoding was more successful when the same person was used for the template run and the test run, the patterns from one person could be used to decode the brain activity of another person.

Then, the researchers made the problem more complex. They showed participants a checkerboard wedge that rotated at 10 degrees a second. Three participants sat for six seven-minute runs on two separate days. Using the same template and test run strategy, the researchers were able to pinpoint the position of the wedge within 26 degrees.

The results are a first step toward the ultimate goal of facilitating communication for people who struggle to express themselves because of cerebral palsy, stroke, or other conditions that result in locked-in syndrome, the researchers say.

“It sounds like a huge jump, from checkerboards to figuring out what words somebody is internally verbalizing to oneself,” Culver says. “But a lot of the principles are the same. The goal is to help people communicate, and what we’ve learned by decoding these visual stimuli is a solid step toward that goal.”

The National Institutes of Health and the McDonnell Foundation funded the work.

Source: Washington University in St. Louis