Scientists have recreated video from brain scans

With the help of generative AI, scientists have been able to create high quality video based on brain scan data.

Scientists from the National University of Singapore and the Chinese University of Hong Kong applied functional magnetic resonance imaging (fMRI) and an AI model Stable Diffusion to develop a model called MinD-Video that produces videos based on brain activity.

On project website you can see the demo. The demo shows the videos the subjects watched and the videos generated by the AI ​​from their brain signal. The two videos are almost indistinguishable from each other. They contain similar scenes and colors.

According to the researchers, MinD-Video is “a two-step process that can bridge the gap between image and video decoding from the brain.” To train the system, the experts used an open dataset that included video clips and fMRI data of the brains of the subjects who watched the videos. The “two-step process” consisted of a trained fMRI decoder and a finely tuned version of Stable Diffusion.

Brain video decoding and video reconstruction

The authors of the study believe that the technology has great potential for applications in various fields, from neuroscience to brain-computer interfaces.

The scientists also noted that three main conclusions can be drawn from the results of the experiment.

1. The visual cortex is a key element of visual perception;
2. The fMRI decoder works in a hierarchical order that starts with structural information and then moves on to more abstract and visual features at deeper levels;
3. The fMRI decoder has evolved at each stage of learning, showing the ability to capture increasingly finer information as learning progresses.

The study is another breakthrough in the field of AI mind reading. Formerly scientists from Osaka University discovered that they can recreate high-quality images based on brain activity using a method that also uses fMRI and Stable Diffusion data.