Playing With Your Thoughts: How VR and BCIs Can Make You Sound Not Crazy. | by Ahnaaf Khan | Nov, 2020

Game Developers Conference

If you’re reading this article, I’m sure you’ve heard of virtual reality. It’s this emerging technology where computer-generated environments can be simulated in real-time. Usually, a VR headset is fitted over your eyes to visually immerse you in virtual reality. There are endless possibilities with virtual meetings, trips, and fully immersive entertainment.

Virtual reality has always been at the forefront of emerging technology. Everyone wants to know, how immersed can you be in a movie or game? What’s the next step to immersing people in truly virtual reality without being distracted by their actual reality? How can you play games with your thoughts and not sound crazy to other people? This is where Brain-Computer Interfaces come in.

No… that would lead to some really shit UX. A Brain-Computer Interface (BCI) in simple terms, is hardware that interprets brain signals and transmits them to a computer. BCIs help with communication between the brain and various machines. Before we talk about what we can do with the VR- BCI intersection, we have to learn more about BCIs. First off, there are 3 types of BCIs:

  1. Invasive
  2. Semi-Invasive
  3. Non-Invasive

Not yet, invasive BCIs are still fairly controversial, especially since people don’t really trust hardware that is going into your head. I’m not going to lie, I wouldn’t either, not even if Neuralink paid me to.

Invasive BCIs work by placing individual electrodes directly in the cortex to measure small groups of neurons.

The base signal from the neurons is taken directly from the cortex and transmitted to a computer without any “noise” or interference. This is the best way to get signals from the brain, as it’s the closest a BCI can be to the brain. These BCIs could also stimulate your brain with neurofeedback (sending electrical signals to your brain) possibly do things like enhance, restore vision, and even upload information to your brain.

It’s literally inside your brain, and while it’s great for that reason, it’s also shady for the same reason. There are a lot of ethical, and security concerns surrounding BCIs in general, (that I won’t cover in this article) especially invasive BCIs. Imagine if someone were to hack into your BCI and fill your brain with ads, or take that data from your brain. There’s a lot of potential for invasive BCIs, but there’s also a lot of downsides, and it isn’t exactly ready for VR yet.

Semi-invasive BCIs are less extreme, but you still need surgery (Craniotomy) to use Electrocorticography (ECoG). Usually, ECoG is only used for medical purposes, but the clinical risk is low. ECoG works by placing electrodes on the surface of the brain, either the dura or the arachnoid. One of the great features of ECoGs is that they have a high spatial resolution.

Think of spatial resolution as the clarity of a picture, higher spatial resolution means there are more pixels per inch and the quality is higher. Even better, ECoGs aren’t affected by any “background noise” like eye movement or muscle movement. Often, there can be a motor in the ECog which applies neurofeedback, which is essentially stimulating your cortex.

The potential for this hardware is massive, especially for the possibility of merging BCI with VR. In this study, it was possible to make patients move a cursor in 2 dimensions with 30 minutes of training, which is insane to think about. Especially in the context of VR, imagine how easy it’s going to control a virtual character with your brain in 5 years.

Even though I’ve been researching this technology, and I know it’s fairly safe, I’m still not sure of how comfortable I am opening up my head. That’s why we use non-invasive methods like Electroencephalography (EEG) for most consumer BCIs. There are more non-invasive methods, but EEG is the most popular since it’s smaller and cheaper. A consumer EEG usually takes the form of a headset like

Muse or Emotiv. An EEG uses electrodes that are placed on your scalp to pick up electrical activity all-around your brain. What’s recorded is the voltage difference between at least 2 different electrodes.

Sadly, EEGs also have their fair share of problems too. Because of its external electrode placement, there’s often a lot of interference in the signal that has to be sorted out and cleaned up. EEGs also have low spatial resolution compared to other more invasive methods. It’s what’s expected when signals have to travel through your skull and skin to get picked up by the EEG.

But, it’s not all that bad, the tradeoff is that EEGs have really high temporal resolution. Temporal resolution is defined as “how closely the measured activity corresponds to the timing of the actual neuronal activity.” In other words, it’s the BCIs latency, when your brain does something, how fast can the BCI react. Higher temporal resolution means that there is low latency (that’s good), and low temporal resolution means the opposite (not good).

This means that EEGs, are perfect for VR! EEGs are extremely useful since for some of them, you can just put it on and go. In practice, EEGs pair really well with VR headsets since you can put electrodes onto the part of the headset which touches your scalp and use that to read your brainwaves. This is why EEGs are the only BCIs being used with VR.

I’m glad you asked, mystery reader! Clearly, if you combine BCI technology with VR, you’ve got a match made in technology heaven. Let’s dive into the design of a VR x BCI game created by Neurosky called “Throw Trucks With Your Mind.” The title basically explains it all, it’s a game about throwing trucks with your mind.

The first step to creating a game is to preprocess user data and extracting the key data. EEG data is messy, and usually, there’s interference from 4 sources:

  1. The EEG equipment itself
  2. Electrical interference not from the EEG or subject (power lines, laptops)
  3. Faulty leads and electrodes
  4. Electrical activity from the heart, eye blinking, eyeball movements, muscle movements in general.


More Posts

Send Us A Message