EFTA02693109

lopscience iopscience.iop.org Home Search Collections Journals About Contact us My lOPscience Fast attainment of computer cursor control with noninvasively acquired brain signals This article has been downloaded from lOPscience. Please scroll down to see the full text article. 2011 J. Neural Eng. 8 036010 (http://iopscience.iop.org/1741-2552/8/3/036010) View the table of contents for this issue, or go to the journal homepage for more Download details: IP Address: 66.44.80.91 The article was downloaded on 16/04/2011 at 04:09 Please note that terms and conditions apply. EFTA_R1_02036188 EFTA02693109 KW PUBLISHING J. Neural Eng. 8 (2011103601019pp) J€ rl ANAL OF NEURAL ENGINF/ROW I 7 II-2560'S I WOW, Fast attainment of computer cursor control with noninvasively acquired brain signals Trent J Bradberry I • • , Rodolphe J Gent1112.3 and Jose L Contreras-Vidal l '23' 5 Fischell Department of Bioengineering, University of Maryland, College Park, MD 20742. USA 2 Department of Kinesiology, University of Maryland. College Park. MI) 20742. USA 'Graduate Program in Neuroscience and Cognitive Science, University of Maryland. College Park. MD 20742. USA E-mail: tretilhGt unid.cdu and pepcutraaurnd cdu Received 7 January 2011 Accepted for publication 18 February 2011 Published 15 April 2011 Online at stacks.iop.org/JNE/8/036010 Abstract Brain-computer interface (BCI) systems are allowing humans and non-human primates to drive prosthetic devices such as computer cursors and artificial arms with just their thoughts. Invasive BCI systems acquire neural signals with intracranial or subdural electrodes, while noninvasive BO systems typically acquire neural signals with scalp electroencephalography (EEG). Some drawbacks of invasive BCI systems are the inherent risks of surgery and gradual degradation of signal integrity. A limitation of noninvasive BCI systems for two-dimensional control of a cursor, in particular those based on sensorimotor rhythms. is the lengthy training time required by users to achieve satisfactory performance. Here we describe a novel approach to continuously decoding imagined movements from EEG signals in a BCI experiment with reduced training time. We demonstrate that, using our noninvasive BCI system and observational learning, subjects were able to accomplish two-dimensional control of a cursor with performance levels comparable to those of invasive BCI systems. Compared to other studies of noninvasive BC1 systems, training time was substantially reduced, requiring only a single session of decoder calibration (-'20 min) and subject practice (--20 min In addition, we used standardized low-resolution brain electromagnetic tomography to reveal that the neural sources that encoded observed cursor movement may implicate a human mirror neuron system. These findings offer the potential to continuously control complex devices such as robotic arms with one's mind without lengthy training or surgery. El Online supplementary data available from cracks itip.orONK/8/036010/mmedia (Some figures in this article are in colour only in the electronic version) I. Introduction Brain—computer interface (BCI) systems may potentially provide movement-impaired persons with the ability to interact with their environment using only their thoughts to control assistive devices such as communication programs Prclecnt address: Matron. Inc.. Reston, Virginia 20190. USA. S Author to whom any correspondence should be addressed. 1741-2560/11/036010.09533.00 and smart artificial anns. Currently the most promising BCI systems rely on neural signals acquired noninvasively with electroencephalography (EEG) (Wolpaw and McFarland 2004) or invasively with electroconicography (ECoG) (Schalk et al 2(x0K) or microelectrode arrays seated into cortical tissue (Hochberg et al 2000. Current noninvasive EEG-based BCI systems for 2D cursor control require subjects to learn how to modulate specific frequency bands of neural activity, i.e. sensorimotor 0 2011 HOP Publishing Lid Printed in the UK EFTA_R1_02036189 EFTA02693110 I. Neural Eng. 8 (2011) 036010 T 1 Bradbeny et al rhythms, to move a cursor to acquire targets (Wolpaw and McFarland 2004). These types of studies based on sensorimotor rhythms require weeks to months of training before satisfactory levels of performance are attained. Relative to EEG signals, the increased signal-to-noise ratio and bandwidth of invasively acquired neural data are commonly thought to be factors that reduce the training time required by users of invasive BCI systems (Schalk a al 2(08). In addition. studies of tetraplegic humans with implanted microelectrode arrays have exclusively demonstrated 213 control of a cursor through imagined natural movement (Hochberg et al 2006. Kim et al 2008). This decoding of imagined natural movement is also a likely factor in reduced training time since neural signals directly correlate with intended actions. However, recently several off-line decoding studies have demonstrated the reconstruction of cursor and hand kinematics from noninvasive magnetoencephalography (MEG) (Bradbeny et al 2009) and EEG (Bradberry et al 2010). The noise and bandwidth limitations of the noninvasively acquired signals did not impede decoding kinematics of natural movement. This finding implies that a noninvasive BCI system based on the decoding method reported in those studies may require little training time. In this study, we sought to investigate the use of the decoding method reported in those off-line studies in an EEG- based BCI system during a single session lasting less than 2 h that required only brief training. We hypothesized that the putative human mirror neuron system (MNS), which predicts and interprets one's own actions and the actions of others (Tkach et al 2008), could be exploited during training by asking subjects to combine motor imagery with observation of a video of cursor movement. In fact., several of the aforementioned invasive studies (Hochberg a al 2006, Kim et al 2(08) successfully demonstrated a similar approach to training. We further hypothesized that a neural decoder could subsequently be built off-line that would predict cursor movement from neural activity, and the decoder could then be used on-line for real-time brain-control of cursor movement with little training time. Furthermore, to provide additional validation of our hypotheses, we sought to examine the involvement of neural regions in encoding cursor velocity during observation of the cursor movement and during tasks requiring a brain-controlled cursor to acquire targets in 213 space. 2. Materials and methods 2.1. Experimental tasks The Institutional Review Board of the University of Maryland at College Park approved the experimental procedure. After giving informed consent, five healthy. right-handed, male subjects performed a three-phase task: calibration, practice and target acquisition. None of the subjects had previously participated in a BO study. In all phases. their EEG signals were acquired while they sat upright in a chair with hands resting in their laps at arm's length away from a computer monitor that displayed a workspace of dimensions 30 cm x 2 Figure 1. Setup of EEG-based BCI experiment. Subjects EEG signals were acquired while sitting in a chair facing a monitor that displayed a cursor and targets (only target acquisition phase). During the calibration phase, subjects observed a computer- controlled cursor to collect data for subsequent initialization of the decoder. In the target acquisition phase (shown in the photo). subjects moved the brain-controlled cursor to acquire targets that appeared at the left, top, right, or bottom of the computer screen. (for a more detailed schematic, see figure s I in the supplementary data, available at Ntachs.iopmrvLINE/8/0360 Illinunethaj 30 cm and a cursor of diameter 1.5 cm (0.20% of workspace) (figure O. Subjects were instructed to remain still and relax their muscles to reduce the introduction of artifacts into the EEG recordings. 2.1.1. Calibration phase. During the calibration phase, subjects were instructed to imagine moving their right arm/finger to track a computer-controlled cursor that moved in two dimensions on the computer screen. The movements of the computer-controlled cursor were generated by replaying a 10 min recording of a pilot subject's brain-controlled cursor movements from one of his practice runs (this pilot subject did not participate as one of the five subjects in this study). Histograms of the horizontal and vertical positions and velocities of the computer-controlled movements indicated approximately uniform coverage of the workspace and biological motion respectively (figure 2). The decoding procedure described in section 2.3 below was subsequently executed (^-10 min of computation time) to calibrate the decoder so that it best mapped the EEG signals to observed horizontal and vertical cursor velocities. During pilot testing, we discovered that asking subjects to visually fixate the center of the workspace while simultaneously tracking the cursor added attentional demands that burdened the subjects and likely compromised the decoding; therefore, we told subjects they were free to move their eyes but to always maintain eye contact and spatial attention with the moving cursor. 2.1.2. Practice phase. During the practice phase. the subjects used the calibrated decoder to attempt to move the cursor with their thoughts in two dimensions as desired EFTA_R1_02036190 EFTA02693111 1. Neural Eng. 8 (2011) 036010 T.1 Bradbeny et al x (B) 2000 1500 WOO 500 S00 -100 0 100 Vela* feral) 2000 1500 1000 500 -100 0 100 Wooly (cm%) Figure 2. Histograms of observed cursor kinematics during the calibration phase. (A) Histograms of horizontal (left) and vertical (right) positions indicated approximately uniform coverage of the workspace. (13) Histograms of horizontal (left) and vertical (right) positions inferred movement: with bell-shaped velocity profiles (although these are more super-Gaussian than typical point-to-point movements). indicative of biological motion. The velocity histograms actually peak near 5000 but were truncated so the shape of the base could be viewed. (without task constraints). They were instructed to determine for themselves how to best control the cursor by exploring the workspace. They were also informed as to where the target locations would be in the target acquisition phase that would follow. Again, they were free to move their eyes. During the initial portion of the practice phase. horizontal and vertical gains were independently adjusted by the investigators to balance cursor speed and to ensure full coverage of the display workspace by the brain-controlled cursor. After the gains were manually adjusted (^-10 min). subjects practiced moving the cursor without task constraints for 10 min. 2.1.3. Target acquisition phase. During the target acquisition phase, subjects were instructed to use their thoughts to move the cursor in two dimensions to reach a peripheral target (1.3% of workspace) that would appear pseudorandomly at the top, bottom, left, or right side of the computer screen. They were informed that if they did not acquire the target within 15 s, a new target would appear, and the trial was considered a failure. Four 10 min runs of target acquisition were performed with a rest interval of 1 min between runs. 2.2. Data acquisition A 64-sensor Electro-Cap was placed on the head according to the extended International 10-20 system with ear-linked reference and used to collect 58 channels of EEG activity. Continuous EEG signals were sampled at 100 Hz and amplified 1000 times via a Synamps I acquisition system and Neuroscan v43 software. Additionally, the EEG signals were band- pass filtered from 0.01 to 30 Hz. Electroocular (EGG) 3 activity was measured with a bipolar sensor montage with sensors attached superior and inferior to the orbital fossa of the right eye for vertical eye movements and to the external canthi for horizontal eye movements. The EEG signals were continuously sent to the BCI2OOO software system (Schalk et al 20lµ) for online processing and storage. BCI2OOO was responsible for moving the cursor based on our decoder function. which we integrated into the open source software system. BC12OOO was also responsible for storing cursor movement data as well as collecting markers of workspace events such as target acquisition. Electromyographic (EMG) signals were amplified and collected at 2000 Hz from two bipolar surface electrodes over the flexor carpi radialis and extensor digitorum muscles of the right forearm using an Aurion ZeroWire system (10-1000 Hz bandwidth, constant electrode gain of 1000). 2.3. Decoding method The decoding method employed in this study has been previously described (Bradberry et a! 2010) so will only briefly be described here. First, a fourth-order, low-pass Butterworth filter with a cutoff frequency of I Hz was applied to the kinematic and EEG data. Very low frequencies have previously been shown to possess kinematic information (Jerbi eta! 2OO7, Walden et al 2(08, Bradberry eta! 2O09), including those from low-pass filtered electrocorticographic signals (the local motor potential, LMP) (Schalk et al 2007). Next, the first-order temporal difference of the EEG data was computed. EFTA_R1_O2O36191 EFTA02693112 I. Neural Eng. 8 (2011) 036010 Ti Bradbeny et al To continuously decode cursor velocity from the EEG signals. a linear decoding model was employed: N L X[I)- - 11= ax + E E bakx Sat - kl (I) st=l 4.0 N L y(t)- yrt — II= ay + E E Nkys„ — k I. (2) n=l 4=0 where x [t] -41- I ) and y — I I are the horizontal and vertical velocities of the cursor at time sample r respectively, N is the number of EEG sensors, L (=I I) is the number of time lags. S„ [I — k] is the temporal difference in voltage measured at EEG sensor n at time lag k. and the a and b variables are the weights obtained through multiple linear regression. Only the most important sensors (N = 34) for velocity reconstruction found in a previous study (Bradberry et al 2010), which excluded the three most frontal sensors, were used for decoding. For the calibration phase, a (10 x 10)-fold cross-validation procedure was employed to assess the reconstruction accuracy of observed cursor velocity from EEG signals. In this procedure, the entire continuous data were divided into 10 parts: 9 parts were used for training, and the remaining part was used for testing. The cross-validation procedure was considered complete when each of the ten combinations of training and testing data were exhausted. and the mean Pearson correlation coefficient (r) between measured and reconstructed kinematics was computed across folds. Prior to computing r. the kinematic signals were smoothed with a fourth-order, low-pass Butterworth filter with a cutoff frequency of I Hz. For the ensuing practice and target acquisition phases. the regression weights (a and b variables) for the cross-validation fold with the highest r were used for online decoding. 2.4. Scalp maps of sensor contributions To graphically assess the relative contributions of scalp regions to the reconstruction of cursor velocity, the decoding procedure described in the section above was run on standardized EEG signals. and the across-subject mean of the magnitude of the best b vectors (from equations (I) and (2)) was projected onto a time series (-110-0 ms in increments of 10 ms) of scalp maps. These spatial renderings of sensor contributions were produced by the topoplot function of EEGLAB (Delorme and Makcig 2004), an open-source MATLAB toolbox for electrophysiological data processing that performs bihannonic spline interpolation (Sandwell 1987) of the sensor values before plotting them. To examine which time lags were the most important for decoding. for each scalp map. the percentage of reconstruction contribution was defined as N E sbr, = 100% x a= (3) E E bLi 17 44;:ky n=l for all i from 0 to Is, where %T; is the percentage of reconstruction contribution for a scalp map at time lag i. 4 2.5. Source estimation with sLORETA To better estimate the sources of cursor velocity encoding. we used standardized low-resolution brain electromagnetic tomography (sLORETA) (Pascual-Marqui 2(102) software version 20081104. Preprocessed (low-pass filtered and differenced) EEG signals from all 34 channels for each subject were fed to sLORETA to estimate current sources. First, r values were computed between the squared time series of each of the 34 sensors with the 6239 time series from the sLORETA solution and then averaged across subjects. Second, the mean of the r values multiplied by the regression weights b (from equations ( I ) and (2)) of their associated sensors were assigned to each voxel. The regression weights had been pulled from the regression solution at the time lag with maximum %T. which had the highest percentage of reconstruction contribution. Third, for visualization purposes, the upper quartile of voxels (r values weighted by b) was set to the value one, and the rest of the r values were set to zero. Finally these binary-thresholded r values were plotted onto a surface model of the brain. 2.6. Eye and muscle activity analysis To assess the contribution of eye activity to decoding, the decoding procedure was executed off-line with channels of standardized vertical and horizontal EOG activity included with the 34 channels of standardized EEG activity. The percent contribution of these cyc channels was then assessed by dividing the absolute value of their regression weights by the sum of the absolute value of all the regression weights. To assess whether muscle activity inadvertently aided cursor control, we cross-correlated EMG signals from flexor and extensor muscles of the right forearm with the .r and y components of cursor velocity over 200 positive and negative lags (-2 s to 2 s in increments of 10 ms). The start of the EMG and EEG/EOG recordings were not synchronized by computer, which is why the cross-correlation of the EMG and EOG signals at different lags was examined as opposed to only the zero-lag correlation. Prior to the cross-correlation, the EMG signals were decimated 20 times after applying a 40 Hz low-pass antialiasing filter; rectified by taking the absolute value: low-pass filtered with a fourth-order, low-pass Butterworth filter at I Hz: and first-order differenced. 3. Results 3.1. Calibrating a neural decoder front observed cursor movement BC1 systems are ultimately intended for movement-impaired persons; therefore, it is imperative that calibration of the neural decoder does not require overt movement. For this reason, we calibrated our previously developed decoder (Bradberry et al 2010) in a manner similar to that described in an invasive BCI study (Hochberg et al 2006) that required only motor imagery during observation of cursor movement. More specifically, during the calibration phase of our study, subjects imagined using their finger to track biologically plausible movement of a computer-controlled cursor for 10 min. and we subsequently EFTA_R1_02038182 EFTA02693113 I. Neural Eng. 8 (2011) 036010 T 1 Bradbeny et al (A) 0.9 05 02 0$ • 0.4 03 0.2 0.I 0 8400•02 44410:13 19.144414 Sublicte Won (B) 4, a +0 10 20 fi r) 40 X Velocity Y Velocity 60 60 10 20 30 40 50 60 lime(s) Flgure 3. EEG decoding accuracy of observed cursor velocity during the calibration phase. (A) We computed the mean standard error (SE) of the decoding accuracies (r values) across crass-validation folds (n = 10) for each subject for x (black) and y (white) cursor velocities. (B) Superimposed reconstructed velocity profiles (red) and actual velocity profiles (black) matched well (data from subject 1). Subject 1 Subject 2 Subject 3 Subject 4 Subject 5 ‘^s Figure 4. Mean hrain-contna led cursor paths. Each colored path is the mean of he length-normalized trials for a single direction (left, top. right, or bottom) across all trials of all runs for a subject. Trials in which subjects did not acquire the target within 15 s were excluded from analysis. Table I. Mean (SE) of the hit rate and median MT for each target of each subject across runs = 4). Left Top Right Bottom Mean (SE) Hat% MT Hu% MT MT Hu% MT Hit% MT Subject I 94 (2) 4.24 66 (8) 5.90 98 (2) 4.62 55 (9) 8.88 78(11) 5.91 (1.05) Subject 2 83 (5) 6.52 96 (4) 4.40 85 (2) 3.76 85 (4) 4.40 87 (3) 4.77 (0.60) Subject 3 84 (9) 4.24 45 (4) 9.96 100 (0) 2.32 67 (9) 6.82 74 (12) 5.83 (1.65) Subject 4 71 (7) 3.40 33 (7) 4.88 65 (6) 8.16 21 (4) 6.68 47 (12) 5.78 (1.04) Subject 5 57 (14) 8.56 100(0) 2.72 60(18) 5.48 100(0) 2.00 79 (12) 4.69 (1.49) Mean (SE) 78 (6) 5.39 (1.06) 68 (13) 5.57 (1.35) 81 (8) 4.87 (1.09) 65 (14) 5.76(1.32) 73 (4) 5.40 (0.27) The median MT, instead of the mean (SE) Mt was computed for each direction of each subject because the MT distributions were skewed. computed the parameters of the decoder (--10 min) based on the cursor velocity and EEG signals. We quantified the accuracy of each subject's calibrated decoder by computing the mean of Pearson's r between actual and reconstructed cursor velocities across ten cross-validation folds (figure 3(A)). The across-subject mean r values for horizontal (s) and vertical (y) velocities were 0.68 and 0.50 respectively. indicating high decoding accuracy. In fact, the accuracy was roughly double that of studies that decoded observed cursor movement from neural activity acquired more focally with intracranial microelectrode arrays (Kim a al 2008. Truccolo a al 2008). Reconstructed velocity profiles also visually matched well with the actual velocity profiles (figure 3(B)). 5 3.2. Applying the neural decoder to move a computer cursor After a subject's neural decoder was calibrated and a —20 min practice phase with the decoder was performed. the subject moved a cursor with his EEG signals to acquire targets that appeared one at a time pseudorandomly at the left, top, right, or bottom of a 21) workspace (see movie I. available at .tacks.iop media). Four 10 min runs of target acquisition were performed with a rest interval of 1 min between runs. The length-normalized cursor paths confirmed the subjects' ability to move from the center to the target (figure 4). For each target of each subject, the target hit rate and movement time (MT) across runs are given in table I. The overall means SE of the hit rate and MT were EFTA_R1_02036193 EFTA02693114 1. Neural Eng. 8 (2011) 036010 T l Bradbeny et al Cursor Movement Observation 77% 101% (A) 62% &I% 52% • • • • • • .122 I I .110 -100 -90 -80 .70 a 8 12 4%, 715.A 122% 65% • 1% 410 • 1 -50 -40 .30 • • 20 -10 ms I -80 55% • 0 Brain-Controlled Cursor Movement ID) • • 1••• (G) • 55% 49% 74% 102% 116% 1 '2), ;,/ • • • • 0 • • I 1 I 1 1 1 % ; • .. - _ zz O S • -110 400 -80 -70 -60 " Xi% r;.•;% C> ig 5 121% 11.1% 9.0% 2% 49% 56% FP) . • • • . • e -50 -40 -30 -20 -10 0 me Figure 5. Neural regions that encoded cursor movement. (A) Scalp sensor contributions to the reconstruction of observed cursor velocity during the calibration phase. Mean (ti = 5) scalp maps of the sensors revealed a network of frontal, central and parietal involvement. In particular, sensors Fl. rcz and CPI-CP4 of the International 10/10 system made the largest contribution. Light and dark colors represent high and low contributors, respectively. Each scalp map with its percentage contribution is displayed above its associated 10 ms time lag. revealing the 12.4% maximal contribution of EEG data at 50 ms in the past. (B) Sources that maximally encoded observed cursor velocity during the calibration phase. We overlaid localized sources (yellow) from 50 ms in the past onto a model of the brain in different orientations to reveal the involvement of the PrG (I), PoG (2), LPM (3). STS (4), and dorsal and ventral LPC (5). (C) Scalp sensor contributions to the brain-controlled cursor velocity during the target acquisition phase. Mean (n = 5) scalp maps of the sensors weights from the subjects' best runs revealed a network that had shifted to involve more central regions than the network of the calibration phase. The scalp maps revealed a 12.1% maximal contribution of EEG data at 50 ms in the past. (D) Sources that maximally encoded brain-controlled cursor velocity during the target acquisition phase. Localized sources (yellow) from 50 ms in the past revealed a substantial involvement of PrG (I) and PoG (2) and some involvement of LPM (3). As in the calibration phase. the STS (4) was involved. In contrast to the calibration phase. the LPC (5) played a minor role, and the IPL (6) played a major role. 73 4% and 5.4O 0.27 s. The change in hit rate across runs is presented for each subject in figure s2 in the supplementary data, available at slacks. ior.org/JNE/8/0360 I Wnimed i 3.3. Neural regions that encoded cursor movement To visualize the contributions of scalp regions and current sources to the reconstruction of cursor velocity, the weights of the decoder were projected onto scalp maps. and sLORETA (Pascual-Marqui 2002) was employed. Scalp maps of sensor contributions to the reconstruction of observed cursor movements in the calibration phase depicted the contributions as a network of frontal, central and parietal regions 6 (figure 5(A)). Within this network, sensors over the frontocentral and primary sensorimotor cortices made the greatest contribution. Concerning time lags. EEG data from 50 ms in the past supplied the most information. In source space at 50 ms in the past, the precentral gyms (PrG), postcentral gyms (PoG), lateral premotor (LPM) cortex, superior temporal sulcus (STS). and dorsal and ventral portions of lateral prefrontal cortex (LPC) played a large role in the encoding of observed cursor velocity (figure 5(B)). Scalp maps of sensor contributions to the brain-controlled cursor velocity were generated from the mean of each subject's best run in the target acquisition phase. They depicted the contributions as having shifted to be more focused within EFTA_R1_02036194 EFTA02693115 1. Neural Eng. 8 (201 1) 036010 Ti Bra bony et al Table 2. Percent contribution of EOG activity to cursor velocity reconstruction. Target acquisition Calibration (best run) X Y X Y Subject 1 0.30 1.58 0.00 0.01 Subject 2 0.00 0.01 0.20 0.18 Subject 3 1.99 9.60 1.54 047 Subject 4 0.00 0.01 94.9 0.04 Subject 5 0.34 0.65 0.06 0.03 Table 3. Mean (SD) across subjects of maximum absolute r values from cross-correlation of forearm flexor and extensor EMG activity with x and y components of cursor velocity. Target acquisition Calibration (best run) X Flexor 0.05 (0.04) 0.05 (0.04) 0.04 (0.02) 0.07 (0.03) Extensor 0.03 (0.02) 0.04 (0.01) 0.07 (0.08) 0.05 (0.04) central regions (figure 5(C)). As in the calibration phase, EEG data from 50 ms in the past supplied the most information. In source space at 50 ms in the past, compared to the calibration phase. a large shift occurred from anterior (frontocentral) to posterior (centroposterior) neural regions. More specifically. there was much less involvement of the LPC, the PrG and PoG exhibited an even more widespread involvement, and the inferior parietal lobule (IPL) made a large contribution (figure 5(D)). 3.4. Eye and muscle contributions A concern in BCI studies is that eye or muscle movements may contaminate EEG signals thereby inadvertently aiding the control of a device/environment that should be controlled by thought-generated neural signals alone. To address this concern, we executed the off-line decoding procedure with channels of vertical and horizontal EOG activity included, and assessed the percent contribution of these eye channels (table 2). The percent contributions were low for the calibration and target acquisition phases except for a very high percent contribution (94.9%) to .r velocity reconstruction for subject 4 during target acquisition. Interestingly. this subject had the lowest decoding accuracy of all participants, suggesting that eye movements disrupted decoding. Furthermore. the fact that hardly any extreme frontal contribution is observed in the scalp maps and sLORETA plots (figure 5) is a testament to the non- contribution of EOG activity to decoding To access whether muscle activity aide) cursor control, we cross-correlated EMG signals from flexor and extensor muscles of the right forearm with the x and y components of cursor velocity to find that all correlations were low (table 3). 7 4. Discussion We report the first EEG-based BCI system that employs continuous decoding of imagined continuous hand movements. Furthermore. we emphasize that the system requires only a single session of decoder calibration (--20 min) and subject practice (-20 min) before subjects can operate it. The off-line decoding results of the calibration phase that used observation of biologically plausible cursor movement were higher than those of invasive BCI studies and may imply, as discussed below, the involvement of a widespread MNS in humans. In the on-line target acquisition phase, subjects controlled a cursor with their EEG signals alone with accuracies comparable to other noninvasive and invasive BCI studies aimed at 2D cursor control. 4.1. Comparison to other BCI studies Our study is the first noninvasive EEG-based BCI study to employ continuous decoding of imagined natural movement. Previous work in EEG-based BCI systems for cursor control required subjects to overcome an initial disconnect between intended movement and neural activity in order to learn how to modulate their sensorimotor rhythms to control the cursor. These studies based on sensorimotor rhythms required weeks to months of training before levels of performance were deemed sufficient for reporting (Wolpaw and McFarland 2004). We believe the fact that we used a decoder based on imagined/observed natural movement, as opposed to neurofeedback training of sensorimotor rhythms, reduced the subject training requirements of our target acquisition phase to only a single brief practice session (-20 mint An ECoG study based on sensorimotor rhythms that had objectives similar to ours also observed that several subjects learned to control a 21) cursor over a short period of time (Schalk et al 2008). Although this ECoG study reduced training time compared to previous EEG studies (Wolpaw and McFarland 2004), some drawbacks included that pre- training time was still taken for the initial selection of control features and for training subjects to first move the cursor in one dimension at a time. We were able to bypass these two pre-training steps. Another drawback of the ECoG study was that all five subjects used overt movement for initial selection of features, and two subjects used overt movement throughout the study. Additionally, the results of our target acquisition phase compare favorably to those in tetraplegic humans that were implanted with intraconical arrays in the arm area of MI (Hochberg et al 2006, Kim et al 2008) even though the performance results of those studies were only computed on data collected weeks to months after training began. Table 4 compares our study to the aforementioned studies. 4.2. Differential encoding of observed and brain-controlled cursor velocity The most notable differences between the regions that encoded for observed cursor velocity and brain-controlled cursor velocity were with the PIG, PoG, IPL and LPC. There was EFTA_R1_02038195 EFTA02693116 I. Neural Eng. 8(2011) 036010 T 1 Bradbeny eta! 7Lbµ 4. Comparison to most relevant human BCI studies of 2D cursor control. Number of subjects Neural data Target size as% of workspace Timeout (s) Movement time (s) Target hit% Wolpaw and McFarland 2004 4 EEG 4.9 I0 1.9 92 Hochberg et al 21)06 Single units NA 7 2.5 85 Kim et at 2(Xnc 2 Single units 1.7 3.1 75 Schalk Hal 20)8 5 ECoG 7 16.8 2.4 63 Present study EEG 1.3 Is 5.4 73 a more widespread contribution from the PrG, PoG and IPL during brain control, which could reflect the increased involvement of imagined motor execution (Miller et al 20 I()) especially since these regions have previously been shown to be engaged in encoding cursor kinematics (Bradberry et al 2009. Jerbi et al 2(x7). The contribution from the LPC was largely attenuated during brain-controlled cursor movements, suggesting a transition out of the imitative learning environment of cursor observation (Vogt et al 2(X07). 43. Implications for a human mirmr neuron system The training by cursor observation in the decoder calibration phase may have engaged the putative human MNS. which predicts and interprets one's own actions and the actions of others (Tkach et at 200%). In fact. neuronal activity acquired from intracortical microelectrode arrays implanted in the dorsal premotor cortex (PMd) and the arm area of the PrG (primary motor cortex, Ml), common sites for BCI-related studies. exhibits qualities of mirror neurons during observation of cursor movements (Cisek and Kalaska 2004, Wahnoun et al 2(x16. Tkach et al 2007). Current electrophysiological correlates of the putative human MNS, as acquired through EEG, are based on modulation of the mu rhythm (8-13 Hz). which exhibits suppression during action observation and action performance (Perry and Bentin 2009). These EEG correlates at the scalp level with high temporal resolution have been reported to be similar to those revealed by neural hemodynamics with high spatial resolution acquired with functional magnetic imaging (IMRI) (Perry and Bentin 2009). Since our examination of cortical sources that encoded observed cursor velocity revealed some regions commonly held to comprise the canonical human MNS (ventral LPM, STS. and LPC) (lacoboni and Dapretto 20(X) and regions reportedly containing mirror neurons related to the task (PMd, MI) (Cisek and Kalaska 20rµ, Wahnoun et al 2($W,, Tkach a al 2007), our method appears to provide detailed temporal and spatial information about the internal representations of both observed and executed actions, which is not provided by the study of mu rhythm dynamics or hemodynamics alone. Our method provides further spatiotcmporal evidence that the MNS is involved during observed cursor movement by indicating the presence of planning activity that peaks at 50 ms in the past, excluding the decoding of passive viewing as an explanation and suggesting predictive decoding informed by forward models (Miall 20)3). 8 5. Conclusion In the near future, it will be important for whole-arm amputees and persons with impaired upper limb movement (e.g., spinal cord injury or stroke) to test our noninvasive BCI system since they are the target population for this assistive technology. Since our findings indicate that calibration of our decoder and initial practice by subjects require a short amount of time in a single session, we expect to avoid burdening patients with lengthy training. Employing our method will also permit future investigations into the putative human MNS, potentially providing further insights into training protocols for BCI systems. Acknowledgments This work was supported by the Office of Naval Research (N000140910126), the National Institutes of Health (P01 HD064653.01), La Fondation Mortice (Paris, France). and the Graduate Research Initiative Fund of the Department of Kinesiology at the University of Maryland. References Bradberry Ti, Gentili R J and Contreras-Vidal J L 2010 Reconstructing three-dimensional hand movements from noninvasive electroencephalographic signals J. Neurosci. 30 31:2 7 Bradherry Ti, Rong F and Contreras-Vidal J L 2009 Decoding center-out hand velocity from MEG signals during visuomotor adaptation Neurointage 47 1691 700 Cisek P and Kalaska J F 2004 Neural correlates of mental rehearsal in dorsal premotor cortex Nature 431 901 h Dclorrne A and Makeig S 2004 EBGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis J. Neurosci. Methods 134 9-21 Hochberg L R, Senuya M D, Friths G M. Mukand J A. Saleh M, Caplan A H. Branner A. Chen D. Penn R I) and Donoghue J P 2006 Neuronal ensemble control of prosthetic devices by a human with tetraplegia Nature 442 164-71 lacoboni M and Dapretto M 2006 The mirror neuron system and the consequences of its dysfunction Nat. Ren Neurosci. 7 942-51 K, Lachaux J P, N'Diaye K. Pantazis D. Leahy R M. Garner° L and Baillet S 2007 Coherent neural representation of hand speed in humans revealed by MEG imaging Proc. Nall Acad. Sc!. USA 104 7676-81 Kim S P, Simeral J D. Hochberg L R. Donoghue J P and Black M 2008 Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia J. Neural Eng. 5 455-76 Miall R C 2003 Connecting mirror neurons and forward models NeuroRepon 14 2 l 1S-7 EFTA_R1_02036196 EFTA02693117 1. Neural Eng. 8(2011) 036010 Ti Bradbeny et al Miller K J. Schalk G. Fetz E E, den Nijs M. Ojemann J G and Rao R P 2010 Conical activity during motor execution, 111040f imagery. and imagery-based online feedback Proc. Nail Acad. Sci. USA 107 4430 5 Pascual-Marqui R D 2002 Standardized low-resolution brain electromagnetic tomography (sLORETA): technical details Methods Find. Erp. Clin. Phannacol. 24 Stipp! I) 5-12 Perry A and Bentin S 2009 Mirror activity in the human brain while observing hand movements: a comparison between EEG desynchronization in the mu-lunge and previous IMRI results Brain Res. 1282 126-32 Sandwell D T 1987 Bihannonic spline interpolation of CEOS-3 and SEASAT altimeter data Geophys. Res. Len. 2 139-42 Schalk G. Kubanekl, Miller KJ, Anderson N R, Leuthardt E C. Ojemann J G. Limbrick D. Moran D. Gerhardt L A and Wolpaw 1 R 2007 Decoding two-dimensional movement trajectories using electrocorticographic signals in humans J. Neural Eng. 4 264 75 Schalk G. McFarland D J. Hinterbergcr T. Birbaumcr N and Wolpaw 1 R 2004 BCl2000: a general-purpose brain-computer interface (BCI) system IEEE Trans. Slimed. En& 51 1031 Schalk G. Miller K J. Anderson N R. Wilson .1 A. Smyth M D. Ojemann J G, Moran D W. Wolpaw J R and Leuthardt E C 2008 Two-dimensional movement control using 9 electroconicographic signals in humans J. Neural Eng. 5 75 84 Tkach D. Reimerl and Hatsopoulos N G 2007 Congruent activity during action and action observation in motor cortex J. Neurosci. 27 13241-50 Tkach D. Reimer .1 and Hatsopoulos N G 2008 Observation-based learning for brain—machine interfaces Cure. Opin. Neurobiol. 185s9 94 Truccolo W. Friehs G M. Donoghue J P and Hochberg L R 2008 Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia J. Neurosci. 28 1 163-78 Vogt S. Buccino G. Wohlschlager A M. Canessa N. Shah N J. -Gilles K. Eickhoff S B, Freund H .1, Rizzolatti G and Fink G R 2007 Prefrontal involvement in imitation learning of hand actions: effects of practice and expertise Neuroimage 37 1371-83 Wahnoun R, He J and Helms Tillery S I 2006 Selection and parameterization of cortical neurons for neuroprosthetic control J. Neural Eng. 3 162-71 Walden S. Preissl H, Demandt E. Braun C. Birbaunter N, Aensen A and Mehring C 2008 Hand movement direction decoded from MEG and EEG J. Neumsci. 2K 1000-s Wolpaw J R and McFarland D .1 2004 Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans Proc. Nall Acad. Sci. USA 101 17x4') EFTA_R1_02036197 EFTA02693118