Researchers at the University of California, Berkeley have succeeded in obtaining and reconstructing a persons thoughts and memories. The result is a crude digital recording of the visual activity occurring in the person’s head putting us one step closer to being able to record thoughts or dreams (and uploading them to YouTube).
Using functional Magnetic Resonance Imaging (fMRI) to obtain the thoughts and computational models to interpret them, UC Berkeley researchers have succeeded in decoding and reconstructing people’s dynamic visual experiences – in this case, watching Hollywood movie trailers. As yet, the technology can only reconstruct movie clips people have already viewed. However, the breakthrough paves the way for reproducing the mental movies inside our heads that no one else sees, such as dreams and memories.
Eventually, practical applications of the technology could include a better understanding of what goes on in the minds of people who cannot communicate verbally, such as stroke victims, coma patients and people with neurodegenerative diseases. It may also lay the groundwork for brain-machine interface so that people with cerebral palsy or paralysis, for example, can guide computers with their minds.
Previously, Gallant and fellow researchers recorded brain activity in the visual cortex while a subject viewed simple black-and-white photographs. They then built a computational model that enabled them to predict with overwhelming accuracy which picture the subject was looking at. In their latest experiment, researchers say they have solved a much more difficult problem by actually decoding brain signals generated by moving pictures such as video and television.
Shinji Nishimoto, lead author of the study, and two other research team members served as test subjects for the experiment. The procedure requires volunteers to remain still inside the MRI scanner for hours at a time. They watched two separate sets of Hollywood movie trailers, while fMRI was used to measure blood flow through the visual cortex, the part of the brain that processes visual information. On the computer, the brain was divided into small, three-dimensional cubes known as volumetric pixels, or “voxels.” Nishimoto explained:
We built a model for each voxel that describes how shape and motion information in the movie is mapped into brain activity.
The brain activity recorded while subjects viewed the first set of clips was fed into a computer program that learned, second by second, to associate visual patterns in the movie with the corresponding brain activity. Brain activity evoked by the second set of clips was used to test the movie reconstruction algorithm. This was done by feeding 18 million seconds of random YouTube videos into the computer program so that it could predict the brain activity that each film clip would most likely evoke in each subject. Finally, the 100 clips that the computer program decided were most similar to the clip that the subject had probably seen were merged to produce a blurry yet continuous reconstruction of the original movie. In other words, the researchers were able to view the test subject’s interpretation of the video clip on a computer screen – they were able to discern the “mental pictures” inside the subject’s head.
