Monitoring, behavior tracking, and capture

Electroencephalography

Electroencephalography records the electrical activity of the brain through electrodes that are fixed to the scalp. Different solutions can be used depending on the experimental conditions. The TMSi Refa (Twente Medical Systems International B.V., Oldenzaal, Netherlands) is a 32-channel ambulatory and stationary system for physiological research that allows to retrieve data from water based sensors. For mobile applications, the StarStim allows to retrieve data from 8 channels wirelessly in hybrid configurations of transcraneal direct current stimulation, and the B-Alert X10 (Advanced Brain Monitoring, Inc., Carlsbad, CA, USA) provides 9 channels of combined mid-line and lateral EEG sites plus an additional channel of electrocardiography, electromyography, or electroculography. In addition, four EPOC headsets (Emotiv Systems, Kwun Tong, Hong Kong).

Eye tracking

Eye tracking technology estimates where individuals direct their eyes along time. These data can provide information about attention, information processing, presence, focus, drowsiness, consciousness, or other mental states. Eye tracking information can be retrieved using different devices. The Tobii TX300 (Tobii Technology AB, Danderyd, Sweden) collects gaze data at 300 Hz yet allows large head movements, thus enabling the study of saccades, correction saccades, fixations, pupil size changes, and blinks. Other portable solutions are also available for different setups, as the low-cost Tobii EyeX (Tobii Technology AB, Danderyd, Sweden) and the Eye Tribe (Copenhagen, Sweden) or the high performance SMI Eye Tracking Glasses 2 (SensoMotoric Instruments GmbH, Teltow, Germany). This device not only allows to record a person’s natural gaze behaviour in real-time in a broad range of daily life applications but also in our CAVE system, for which the Arrington Scene Camera Eye Tracking (Arrington Research Inc, Scottsdale, AZ, USA).

Electromyography

The TMSi Refa can provide surface electromyography data, a non-invasive technique that estimates the electrical activity of a muscle placing electrodes on (not into) the skin overlying it.

Galvanic skin response

Galvanic skin response or skin conductance represents the changes in the electrical properties of the skin, which can illustrate the autonomic nerve responses as a parameter of the sweat gland function. This physiological measure can be estimated using the TMSi Refa , which uses two electrodes that are attached to a finger, the Q-sensor (Affectiva, Waltham, MA, USA), a wireless Bluetooth bracelet, or the E4 wristband (Empatica, Milan, Italy).

Electrocardiography

The electro-physiological measurement of the heart muscle signal or electrocardiography can be obtained using the TMSi Refa. Besides the heart rate variability, the system also allows to extract the size and position of the chambers and the presence and place of any damage to the heart. Measures of the blood volume pulse, from which heart rate, heart rate variability, and other cardiovascular features may be derived can be estimated using the photoplethysmography sensor of the E4 wristband.

Skin temperature

The skin temperature can be measured via a thermic sensor using the TMSi Refa or via infrared thermopile using the E4 wristband.

Oxygen saturation

The fraction of oxygen-saturated hemoglobin relative to total hemoglobin in the blood can be provided by the TMSi Refa.

Motion tracking

Human movements can be detected by many different devices. The IGS150 (Synertial, Brighton, United Kingdom) is a motion capture solution that estimates de orientation of 15 inertial sensors made up of 9 axes of accelerometers, magnetometers and gyroscopes combined together. Two IGS-Cobra gloves with 7 sensors are also available for hand tracking capture. Conventional tracking systems detect the position of markers, as the optical solution using OptiTrack V100:R2 cameras (NaturalPoint, Corvallis, OR, USA), or sensors, as the electromagnetic G4 (Polhemus, Colchester, VT, USA), which are usually attached to specific body parts to determine their position in a 3D environment. Recent advances in computer vision have made human pose recognition from depth images reality. Even though it cannot be considered as a tracking system in the literal sense, the skeleton tracking provides the 3D position of many body joints from depth information estimated by depth sensors, such as the Kinect and Kinect v2 (Microsoft, Redmond, WA, USA), the Intel RealSense F200 (Creative Technology Ltd, Singapore), and the Structure Sensor (Occipital Inc, Boulder, CO, USA).

Other devices for natural interaction are also available. The Leap Motion (Leap Motion Inc, San Francisco, CA, USA) allows user to interact with finger and hand movements. The Razer Hydra (Razer, Irvine, CA, USA) estimates arm movements by using a weak magnetic field to detect the absolute position and orientation of the controllers with high precision. In contrast to these devices, the Myo armband (Thalmic Labs, Kitchener, ON, Canada) allows users to interact with gestures by detecting the surface electrical activity of the wrist muscles.

Capture

Full 360º video recordings can be obtained combining 14 GoPro Hero 3 (360Heros, Olean, NY, USA). Natural audio can be captured using the binaural audio microphone 3Dio Free Space (J.Tesori Co, Tokyo, Japan) and the Zoom H4n handy recorder (Zoom Corporation, Tokyo, Japan).