Skip to main content

Human motor decoding from neural signals: a review

Abstract

Many people suffer from movement disability due to amputation or neurological diseases. Fortunately, with modern neurotechnology now it is possible to intercept motor control signals at various points along the neural transduction pathway and use that to drive external devices for communication or control. Here we will review the latest developments in human motor decoding. We reviewed the various strategies to decode motor intention from human and their respective advantages and challenges. Neural control signals can be intercepted at various points in the neural signal transduction pathway, including the brain (electroencephalography, electrocorticography, intracortical recordings), the nerves (peripheral nerve recordings) and the muscles (electromyography). We systematically discussed the sites of signal acquisition, available neural features, signal processing techniques and decoding algorithms in each of these potential interception points. Examples of applications and the current state-of-the-art performance were also reviewed. Although great strides have been made in human motor decoding, we are still far away from achieving naturalistic and dexterous control like our native limbs. Concerted efforts from material scientists, electrical engineers, and healthcare professionals are needed to further advance the field and make the technology widely available in clinical use.

Background

Every year, it is estimated that more than 180,000 people undergo some form of limb amputation in the United States alone [1]. In 1996, a national survey revealed that there are 1.2 million people living with limb loss [2]. The figure is expected to be more than tripled to 3.6 million by year 2050 [1]. Besides amputations, various neurological disorders or injuries will also affect one’s movement ability. Examples include spinal cord injury, stroke, amyotrophic lateral sclerosis, etc. Patients suffering from these conditions lose volitional movement control even though their limbs are still intact. No matter if it is amputation or neurological disorder, affected patients have their everyday life and work significantly disrupted. Some may be forced to give up their original jobs, while some may even lose the ability to take care of themselves entirely.

Fortunately, although part of the signal transduction pathway from higher cortical centers to muscles have been severed in those aforementioned conditions, in most of the cases we can still exploit the remaining parts to capture the movement intention of the subject. For amputation, the neurological pathway above the nerve stump is mostly intact. For neurological disorders and injuries, depending on the site of the lesion, usually upper stream structures are still intact and functioning. With modern neural interfacing technology, signal processing and machine learning algorithms, it is now possible to decode those motor intentions and use it to either replace the lost function (e.g. through a prosthesis) or to help rehabilitation (e.g. in stroke [3, 4]).

The signal for movement control can be intercepted at various points along the neural transduction pathway. Each of these points exhibits different features and poses unique advantages and challenges. Some of the methods are more invasive (e.g. intracortical recording) but also more versatile because they intercept neural signals at the upmost stream, so they are less reliant on the presence of residue functions. However, some others (e.g. surface electromyogram) while are less invasive, rely heavily on the presence of downstream functional structures and thus any upstream damages undermine their performance. Ultimately, the choice of signal modality to decode from depends on the location, type, and severity of the lesion. In this review, we will discuss the various opportunities available to decode motor intention from human subject at different locations along the motor control pathway. It is our hope that this comprehensive information can help make the most effective clinical decision on how to help the patients.

In this review, we will mainly focus on the decoding of motor intention on human subjects. Although animal studies are an very important and indispensable part of motor decoding research, the application on human subjects is the ultimate goal. Clinical trials on patients may introduce additional and non-negligible challenges to the system and experimental design. For example, in amputees or paralyzed subjects the ground-truth for limb movement is usually unavailable. Special considerations must be incorporated into the experimental design to work around this limitation. Furthermore, although some methods may be working very well on animal studies, their translation into human use may not be straightforward due to safety concerns or surgical difficulties. Therefore, a focus on human studies will allow us to have a more realistic expectation of the current state-of-the-art performance in the field. This knowledge can then in-turn better inform the decision choosing between risk and benefit of a decoding strategy.

Main text

Neurophysiology of motor control

To decode the motor intention of human subject, it is useful to first understand the natural neurophysiology of motor control, so that we may know where to intercept the control signal and what kind of signal feature that we may encounter.

Motor controls in the human body begins at the frontal and posterior parietal cortex (PPC) [5, 6]. These areas carry out high-level, abstract thinking to determine what actions to take in a given situation [7]. For example, when confronted with a player from the opposing team, a soccer player may need to decide whether to dribble, shoot or pass the ball to his teammate. The choice of the best action depends on the location of the player, the opponent and the ball. It also depends on the current joint angles of the knees and ankles in relation to the ball. The PPC receives input from the somatosensory cortex to get information on the current state of the body. It also has extensive interconnection with the prefrontal cortex, which is responsible for abstract strategic thoughts. The prefrontal cortex may need to consider other factors beside the sensory information about the current environment. For example, how skillful is the opponent compared to myself? What is the existing team strategy at the current state of the game, should I play more aggressively or defensively? The combination of sensory information, past experience, and strategic decision in the frontal and posterior parietal cortex determine what sequences of action to take.

The planning of the action sequence is then carried out by the premotor area (PMA) and the supplementary motor area (SMA), both located in Brodmann area 6 of the cortex. Stimulation in area 6 is known to elicit complex action sequence and intracortical recording in the PMA shows that it is activated around 1 second before movement and stops shortly after the movement is initiated [8]. Some neurons in the PMA also appear to be tuned to the direction of movement, with some of them only be activated when the hand move in one direction but not in the other.

After a sequence of action is planned in PMA or SMA, it requires input from the basal ganglia to actually initiate the movement. The basal ganglia contains the direct and indirect pathway [911]. The direct pathway helps select a particular action to initiate, while the indirect pathway filters out other inappropriate motor programs. In the direct pathway, the striatum (putamen and caudate) receives input from the cerebral cortex and inhibits the internal globus pallidus (GPi). In the resting state, GPi is spontaneously activated and inhibits the oral part of the ventral lateral nucleus (VLo) of the thalamus. Thus, inhibition of GPi will enhance the activity of VLo, which in turn excites the SMA. In the indirect pathway, the striatum excites GPi through the subthalamus nucleus (STN), which then suppresses VLo activity and in turn inhibits SMA. In some neurological disorder like Parkinson’s disease, deficit in the ability to activate the direct pathway will lead to difficulty in initiating a movement (i.e. bradykinesia), while deficit in the indirect pathway will lead to uncontrolled movement in the resting state (i.e. resting tremor).

After the basal ganglia helps filter out unwanted motor programs and focus on the selected programs, the primary motor cortex (M1) will be responsible for their low-level executions [12]. In the layer V of M1, there are population of large neurons pyramidal in shape that project their axon connections down the spinal cord through the corticospinal track. These axons connect with motor neurons in the spinal cord monosynaptically to activate muscles fibers. They also connect with inhibitory interneurons in the spinal cord to inhibit antagonistic muscles. This structure allows one single pyramidal cell to generate coordinated movement in multiple muscle groups.

Motor neurons in the spinal cord receive inputs from the M1 pyramidal cells through the corticospinal track [13]. They also receive the input indirectly from the motor cortex and cerebellum through the rubrospinal track, routed via the red nucleus in the midbrain. Although its functions is well established in lower mammal, the functions of the rubrospinal track in human appears to be rudimentary. Motor neurons in the ventral horn of the spinal cord bundle together to form the ventral root, which exits the spinal cord and joints with the dorsal root to form a mixed spinal nerve. The spinal nerve further branches out to smaller nerve fibers that innervate various muscles of the body. One motor neuron may supply multiple muscle fibers, collectively known as one motor unit. A muscle consists of multiple muscle fibers, grouped into motor units of various sizes, each of which may be supplied by different motor neurons. In large muscles such as those in the leg, one motor neuron may supply hundreds of muscle fibers. In smaller muscles, such as those in the fingers, one motor neuron may only supply 2 or 3 muscle fibers, enabling fine movement control.

The motor control pathway of the human body goes from the high level associative area of the brain, mediated by the motor cortex, through the spinal cord to the individual muscle fibers. Each of the stages plays a different role and uses different mechanisms to ensure that a movement is carried out in a coordinated and smooth manner. Each of these stages also offers different signal modalities and features that can be exploited for motor decoding. We will now discuss these features and strategies to utilize them in details below. An overview showing the motor control pathway and various ways to intercept the control signal is shown in (Fig. 1).

Fig. 1
figure 1

Overview of various ways to intercept motor control signals. Motor control signal is relayed from the primary motor cortex of the brain, via the spinal cord and peripheral nerve, to the muscle fibers. The control signal can be intercepted at various points using different techniques. Electroencephalography (EEG) captures the superimposed electrical fields generated by neural activity on the surface of of the scalp. Electrocorticography (ECoG) measures activity underneath the scalp on the surface of the brain. Intracortical recordings penetrate into the brain tissue to acquire multi- and single-unit activities. Electrodes can also be placed on the peripheral nerve to monitor the low level signal used to drive muscle contraction. Finally, electromyograph (EMG) can also be used to monitor the activity of the muscle directly (the figure contains elements of images adapted from Patrick J. Lynch and Carl Fredrik under Creative Commons Attribution License)

Cortical decoding of limb movements

All volitional motor controls originate from the brain. The motor cortex of the brain plays an especially important role in planning and executing motor commands. For some patients, the brain is the only site where motor intention can be captured because they have lost motor functions in all their extremities (e.g. in tetraplegic patients). Therefore, many efforts have been invested in cortical decoding.

Electroencephalography (EEG)

EEG is the measurement of weak electrical signals from the brain on the surface of the scalp. Its origin is believed to be the summation of postsynaptic potentials of excitable neural tissues in the brain [14]. The skull, dura and cerebrospinal fluid between the brain and the EEG electrodes attenuate the electrical signal significantly, thus the EEG signal is very weak, typically under 150 μV. Those structures also act like temporal low-pass filters, limiting the useful bandwidth of the EEG signal to be below 100 Hz [15]. Furthermore, due to the volume conduction effect of current sources in the head, the effect of a single current source spreads to several electrodes. The result is a spatial low-passing of the original signal, leading to a “smearing” of the signal source and reduction in the spatial resolution. Thus most EEG setups for motor decoding only involve 64 or 128 electrodes. Setups with higher than 128 electrodes are uncommon.

EEG signal is traditionally separated into several frequency bands (delta: 0 – 4 Hz, theta: 4 – 7.5Hz, alpha: 8 – 13Hz, beta: 13 – 30Hz, gamma: 30 – 100Hz). Of particular importance to motor decoding is the brain oscillation in the alpha band over the motor and somatosensory cortex, also known as the μ-rhythm [16, 17]. It has been observed that there is a decrease of the signal power in the 8 – 13 Hz band when a subject is carrying out actual or even imagined movement [18, 19]. Similar observations can also be found in the lower beta band (12 – 22Hz). Although some components of the beta band oscillation may be harmonics of the alpha band signals, the common consensus now is that they are independent signal features due to having different topographic and timing characteristics [18, 20]. The mu-rhythm tends to focus on the bi-lateral sensorimotor area while the beta rhythm concentrates mainly on the vertex. Collectively, the modulation of the signal band power over the sensorimotor area is called sensorimotor rhythm (SMR).

This decrease of band power coinciding with an event is called event-related desynchronization (ERD). The opposite is called event-related synchronization (ERS), which is the increase of band power coinciding with an event. ERD/ERS is typically calculated with respect to a reference period, usually when the subject is wakefully relaxed and not doing any task [21]:

$$ERD = \frac{R-A}{R} \times 100\% $$

where R is the band power during the reference period and A is that during the time period of interest. An example of ERD topography during motor imagery is shown in (Fig. 2).

Fig. 2
figure 2

Examples of EEG features in motor decoding. EEG features from one of the subject from the BCI Competition IV 2a dataset [214]. a The time course of the change in band power of the EEG signal filtered between 8-12Hz, in left hand and right hand motor imagery, compared to a reference period (0-3s). The shaded regions show the standard deviation of the changes across different trials. The experimental paradigm is also shown below. b The frequency spectrum of the EEG signal during the fixation and motor imagery (c) the topography of the ERD/ERS distribution in different types of motor imagery

The ERD topography during movement displays an evolving pattern over time [21]. ERD usually starts around 2 s before actual movement, concentrating on the contralateral sensorimotor area, then spreads to the ipsilateral side and becomes bilaterally symmetrical just before the start of movement. After the movement terminates, there is an increase of beta band power (i.e. ERS) around the contralateral sensorimotor area [2123], also known as the “beta rebound”. The occurrence of beta rebound coincides with reduction in corticospinal excitability [24], suggesting the rebound may be related to the deactivation of the motor cortex after a movement terminates. Beta rebound occurs in actual as well as in imagined movements. An example of the beta rebound can be observed in (Fig 2a).

Different kinds of motor imagery (MI) produce different topograpies of ERD and hence are useful for decoding the motor intention of the subject. For example, imaging moving one’s hand will elicit ERD near the hand area of the motor cortex, which is in the more lateral position. On the other hand, imaging a foot movement will elicit ERD near the foot area in some of the subjects, which is closer to the sagittal line [25], as can be observed in (Fig. 2c). The beta rebound after MI also displays a similar somatotopic pattern [22]. Simultaneous ERD and ERS on different parts of the brain is also evident in some of the subjects. For example, some subjects showed ERD in the hand area and ERS in the foot area during a voluntary hand movement, and vice versa during a foot movement[22]. ERD may represent an activation of the cortical area controlling the motion while an ERS may represent an inhibition of other unintended movements. As we recall from the neurophysiology of motor control, the indirect pathway of basal ganglia contains mechanisms to suppress the thalamic activation to SMA to filter out unintended movements. There are characteristic patterns of ERD/ERS during different actual and imagined movements, thus by looking into those patterns we can detect and distinguish the motor intention of different body parts.

The most reactive frequency band at which ERD/ERS occurs may be specific for each subject and even for the type of motor imagery, and its topography may vary slightly across different EEG preparations. Therefore, signal processing and machine learning techniques are usually employed to adapt to the signal features of the subjects automatically.

One of the most important signal processing step in SMR-based motor decoding is the estimation of signal power in the frequency range of choice, typically in the alpha (8–12 Hz) and beta (12–30 Hz) band. There are many methods to achieve this. One of the simplest and most computational efficient method is band-pass filtering [3, 26]. The EEG signal is first band-pass filtered in the frequency band of interest, then the sum of the square of the signal is then taken as the power of the signal in the chosen frequency band. Sum-of-the-square is equivalent to the variance of the signal, so usually the variance of the signal is used instead. After taking the variance, a log-transform is commonly employed. The log-transform can serve two purposes. First, it transforms skewed data to make them more conforming to the normal distribution [27], which may help improve performance in some classification algorithms. Second, the log-transform emphasizes the relative change of the signal rather than the absolute difference (e.g. log(110)−log(100)=log(1100)−log(1000)), so it can perform an implicit normalization of the signal and improve the performance of the classifier.

One of the major drawbacks of the simple band-pass filtering approach is that it may be difficult to choose the best frequency band to perform the filter, as each patient has their own specific reactive band. To overcome this limitation, the adaptive auto-regressive (AAR) model is another commonly employed technique [2831]. It models the signal at current time point as a linear combination of previous p points:

$$Y_{t} = a_{1,t} Y_{t-1} + a_{2,t} Y_{t-2} + \dots + a_{p,t} Y_{t-p} + X_{t} $$

where Yt is the signal, Xt is the residue white noise and ap,t the autoregressive coefficients. The core difference with the traditional AR model is that in the AAR model, the coefficients ap,t are dependent on time and are calculated for each signal time point using recursive least square [32]. AAR coefficients from multiple electrodes are then concatenated together to form the feature vector used by a classification system. AAR coefficients can be seen as the impulse response of a system and so it contains information about the frequency spectrum of the modeled signal. Compared to the traditional band-pass filtering, spectrum estimation using AAR can be more robust against noise. One can also specify the number of spectrum peaks based on domain knowledge (each peak requires two coefficients). Another advantage is that there is no need to choose a subject-specific frequency band beforehand as all model coefficients are used for classification. Another way to choose the subject-specific frequency band automatically is to use a filter bank that consists of multiple band-pass filters in different frequencies. After filtering, the most informative frequency band and channels are then selected using some performance metrics, e.g. whether deleting those feature will lead to a reversal of the classification label [33, 34].

Due to the volume conduction problem in the human head, a single current source often appears to be “smeared” across several EEG electrodes. Spatial filtering is usually employed to improve the spatial resolution of the EEG signal. Popular spatial filters include the common average reference (CAR) and surface Laplacian [35]. These methods re-reference the signals by subtracting the voltage at each electrode from the average (as in CAR) or from its neighbors (as in surface Laplacian).

$$V_{j}^{CAR} = V_{j} - \frac{1}{N} \sum_{N}^{k=1} V_{k} $$
$$V_{j}^{LAP} = V_{j} - \frac{1}{n}\sum_{k \in S_{j}} V_{k} $$

where V is the signal voltage, N is the total number of electrodes, n the number of neighboring electrodes, and S is the set of neighboring electrodes in surface Laplacian (LAP).

These filters enhance the focal activity by acting like a high-pass spatial filter. There are also other more advanced spatial filters proposed. For example, the popular common spatial pattern (CSP) [36, 37] works by finding a projection of the electrode voltage such that the differences in variance between two classes are maximized. A further variation of the method is to add in frequency information by filtering the signal by a set of filter bands and then calculate the CSP for each, and finally select the most informative feature through a mutual information criterion [38].

The performance of EEG-based motor decoding has been improving steadily over the years. While earlier studies can only distinguish between discrete types of motor imagery [39], recent studies have already achieve 2D [40] and 3D control [4143]. Some of the latest studies even demonstrate that it is possible to decode different movements in the same limb [44, 45] or even individual finger movements [46].

Besides being used to replace the lost functions, EEG-based motor decoding can also be used a tool for rehabilitation. For example, it can be used to control a robotic hand to assist in active hand training in post-stroke rehabilitation [4, 47, 48]. This application of motor decoding as a tool for training is a very promising area, as it can potentially extend its use to a wider population.

Electrocorticogram (ECoG)

ECoG is the measurement of the electrical signals from the brain on top of the dura, but underneath the skull. ECoG measurement is commonly performed before an epilepsy surgery to delineate the epileptogenic area and identify important cortical regions to avoid during a resection [49]. ECoG signal is not affected by the skull and thus tends to have a higher temporal and spatial resolution than EEG. It also has a larger bandwidth (0 to 500 Hz) [50, 51] and higher amplitude (maximum 500 μV [52]). Therefore, generally ECoG has a higher signal-to-noise ratio than EEG although it is also more invasive.

ECoG and EEG likely arise from the same underlying neural mechanisms therefore they share many similarities with each other. Howevers, there are two major signal features in motor decoding that are unique to ECoG and are specifically exploited. The first is the change of signal band power in the high gamma band (≥75Hz). Many studies have suggested that the high gamma band contains more informative features for motor decoding compared to the alpha and beta band, which are typically used in EEG decoding [5357]. Interestingly, the high gamma band tends to increase during movement, unlike the alpha and beta band, which typically show desynchronization (i.e. decrease in power). Therefore, high gamma power may be produced by a different neural mechanism than the one that produces the alpha and beta desynchronization.

Another unique feature is the low-frequency amplitude modulation of the raw ECoG signal, coined as the Local Motor Potential (LMP) by Schalk et al. [30, 51]. It was found that the envelop of the raw ECoG shows a striking correlation to the movement trajectory of the human hand, as measured by a joystick. The amplitude also shows a cosine or sine tuning in relation to the movement direction, similar to what have been observed in intra-cortical recordings. Since this discovery, many group have incorporated the LMP into ECoG motor decoding in addition to other high frequency features (e.g. [53, 56, 58, 59]). The LMP is a very low frequency component (2-3 Hz) of the raw ECoG signal. It is usually extracted by Guassian low-pass filter, running average [30, 53, 59], or the Savitzky-Golay filter [58, 60, 61].

Due to the robustness of the LMP signal, usually a simple linear regression is sufficient to decode the motor intention in many of the previous studies (e.g. [51, 62, 63]), although a feature selection or regulation step may be needed to first remove the uninformative features. A recent study using deep neural network also show promises [64], however its improvement compared to classical techniques is not always significant.

Because ECoG has a better resolution and higher signal-to-noise ratio, it tends to produce better and finer results than EEG in motor decoding. Beside decoding the movement of different body parts as in EEG [65, 66], different hand gestures can also be distinguished [56, 67]. Using the LMP in addition to frequency features, position and velocity of 2D arm movement can also be decoded from ECoG signals [30, 51, 58]. Subsequent studies even demonstrate that continuous finger positions can also be decoded [54, 59, 61, 63, 64, 68]. The correlation coefficient between the predicted and actual finger movement can reach from 0.4 to 0.7 in some of the recent studies [61, 64].

The large majority of studies in ECoG motor decoding are performed on epilepsy patients without a specific movement disorder or limb injury. However, one of the strongest motivation for motor decoding is that it can compensate the lost motor function of a patient. Given that the brain may re-organize due to disease or injury, it is vitally important that the decoding experiments be repeated on those patient population as well to see if similar decoding performance can be achieved. There are only a few studies to try ECoG motor decoding in stroke patients [57, 69] and paralyzed subjects [70], but the results are encouraging.

Intra-cortical recordings

Penetration into the cortical tissue offers the closest proximity to the neurons and produces the most precise signal. Since the discovery of the directional tuning property of the neurons in the motor cortex [71], a lot of studies have been trying to decode motor intention from intracortical recordings, first in non-human primate (NHP), then in human subjects in recent years. Our review will focus on intracortical decoding in human as it presents some unique challenges compared to NHP, and it is also where the technology will ultimately be applied.

Penetrating electrodes for motor decoding are usually implanted into the primary motor area of the brain. There is a structure in the precentral gyrus resembling a “knob” that houses a majority of the neurons responsible for motor hand function [72]. This “motor hand knob” is typically used as the target for electrode implantation (e.g. in [7377]). Another potential target for implantation is the posterior parietal cortex (PPC). Although PPC has long been proposed to play an important role in the associative functions, in recent years more and more evidence suggests that it also encodes the high-level motor intention of the subject [78]. A recent study suggests that the goal and trajectory of the movement can be decoded from neural activities in human PPC [79].

One important property exhibited by the neurons in the M1 is directional tuning. Some of the neurons are broadly tuned to a particular direction. They discharge the strongest when the movement is in their preferred direction, but they will also discharge less vigorously when the movement is in other directions. Their firing rates represent the length of their preferred direction vector. When the vectors of those neurons are summed together, it indicates the final direction of the movement. This population encoding of movement is a striking property of the nervous system. Similar analog of population encoding can also be found in the super colliculus representing the direction of eye movement [80]. An example showing the directional tuning property of M1 in a non-human primate is shown in (Fig. 3).

Fig. 3
figure 3

Examples of directional tuning in intra-cortical signals. Diagrams showing the directional tuning properties of the neurons in non-human primate M1 from the data in [215, 216]. a Spike raster plots from one of the neurons (Neuron 31). Each plot shows the spike timing of the neuron aligned to the time point (t=0) at which the movement speed of the hand exceeds a pre-defined threshold. Each dot in the plot represents an action potential. Different plots indicates the neuronal activity when the hand is moving in different directions. b The von Mises tuning curve of some of the representative neurons. c The preferred direction of all the neurons. The length of the vector represents the modulation depth of the neuron, here defined as the magnitude of the tuning curve divided by the angle between the maximum and minimum point on the curve

Currently, the only FDA-approved, commercially available microelectrode array for temporary (<30 days) intracortical recordings is the Neuroport System (Blackrock Microsystem, Inc, USA). As a result, majority of the work on human intracortical decoding are performed on that platform. Other intracortical electrodes do exist but they are either mainly for acute intraoperative monitoring (e.g. Spencer Depth Electrode, Ad-Tech; NeuroProbes, Alpha Omega Engineering Ltd; microTargeting electrodes, FHC), or for EEG applications (e.g. DIXI Medical Microdeep Depth Electrodes).

The activities of the neurons in the implanted site are represented by their action potentials, which manifest as spikes in extracellular recordings. Therefore, detecting the occurrence of a spike is often the first step in intracortical signal processing. There are many methods for spike detection [81, 82]. The signal is typically first band-passed filtered in the spike frequency band (e.g. 300-5000Hz), then various methods are used to transform the filtered signal to improve its signal-to-noise ratio (SNR). A detection threshold is then calculated to distinguish spikes from background noise. One of the most common spike detection methods is to use the root-mean-square of the signal

$$Thres = C* \sqrt{\frac{1}{N}\sum_{n=1}^{N}x[\!n]^{2}} $$

where Thres represents the detection threshold above which a signal time point is considered belonging to a spike. However, the RMS value may be easily contaminated by artifacts, so another way is to use the median to set the detection threshold [83]

$$ \sigma = median\left(\frac{|x|}{0.6745}\right) $$
$$ Thres = 4*\sigma $$

The non-linear energy operator is also another popular method [83]. It first transforms the signal such that the high frequency component is amplified to improve the SNR.

$$\psi(x[n]) = x[\!n]^{2}-x[\!n+1]x[\!n-1] $$
$$Thres = C\frac{1}{N}\sum_{n=1}^{N} \psi[x(n)] $$

Other more advanced techniques like continuous wavelet transform [84] and EC-PC spike detection [82] can offer a better accuracy but at a higher computational cost. Although there are a lot of ways to detect spike accurately offline, not every one of them are fast enough to be used in real-time. Therefore in online decoding the choices are usually limited to the simpler algorithms. Manually setting a threshold by an operator still remains one of the most commonly used method. Another popular method in online decoding is the RMS method due to its high efficiency.

An electrode may record signals from multiple neurons nearby. Isolating the activity of a single neuron (i.e. signal-unit activity) from this multi-unit activity usually leads to better results in motor decoding. This process is called spike sorting. There is a large body of literature on spike sorting that cannot be exhausted here. Interested readers are encouraged to consult other excellent reviews [8587]. In practice, the most popular way to do online, real-time spike sorting is via template matching. A set of spike templates are collected during a period of initial recording, then subsequent spikes are classified by comparing their similarity with the templates. However, it may not be really necessary, or may even degrade the decoding result, to do online spike sorting. The spike clusters obtained from recordings may not be stable across different sessions of experiments. The total number of single units sorted from recording may change from sessions to sessions [79]. Thus a decoder trained on some sorted spikes may not work well on future sessions. Spike sorting may also introduce additional latency in online decoding, as accurate spike sorting is a computationally expensive process. In fact, many recent decoding studies do not use spike sorting at all, e.g. [79, 8894].

A decoding algorithm reconstructs motor kinematics from neural activity. Since the discovery of the directional tuning property of motor neurons, one of the earliest decoding algorithm for intracortical spike signal is the population vector algorithm[95, 96]. In its simplest form, the firing rate of a neuron can be related to its preferred direction by

$$f = f_{0} +f_{max} cos(\theta-\theta_{p}) $$

where f is the neural firing rate, f0 and fmax are regression constants and θ and θp are the current and preferred direction respectively. However, for cosine function the width of the modulation is fixed. A more flexible tuning function that allows adjustable width of the modulation is the von Mises tuning function [97]:

$$f = b+k \; exp(\kappa cos(\theta-\mu)) $$

where b, k, κ,μ are the regression constants, and θ is the current movement direction. When μ=θ, the function will be at maximum, so μ can also be interpreted as the preferred direction of the neuron. Examples of the von Mises tuning curves are shown in (Fig. 3b).

The preferred directions of each of the neurons then can be summed together to predict the target direction [96].

$$P(M) = \sum^{N}_{i=1} w_{i}(M) C_{i} $$

where Ci is the preferred direction for the i-th neuron, and wi(M) is the weighting function combining the contributions of each neuron in direction M to the final population vector. However, this method requires a large number of neurons to be accurate and may lead to error if the distribution of the preferred direction is not uniform [98]. For example in (Fig. 3c), we can see that the preferred directions are not distributed evenly. For this reason, a simple linear regression scheme is usually employed instead in recent studies [73],

$$\mathbf{u} = \mathbf{Rf} = \mathbf{R} (\mathbf{R}^{T} \mathbf{R})^{-1}\mathbf{R}^{T} \mathbf{k} $$

where R is the neural response matrix (e.g. firing rate), f is the linear filter (or the regression constants) and k is the motor kinematic values (e.g. joint angles or cursor positions). It has been suggested that this regression scheme can provide more accurate prediction compared to the summation of preferred direction vectors, especially when those vectors are not uniformly distributed [98].

In recent years, the Kalman filter is usually employed instead of the simple linear regression (e.g. in [7577, 99, 100]). The Kalman filter incorporates the information both from an internal process model and actual measurement to estimate the states of a system [101]. A Kalman gain variable is used to determine the “mixing weight” of the model and measurements. If the model is more accurate, then it will trust the model more. The same goes for the measurement. Kalman filter is especially useful if the states are not directly observable or if the measurement is very noisy, which are often both true in motor decoding. In motor decoding, the subjects usually lost their limb or ability to move, therefore the internal state (e.g. motor intention) of the system is not directly observable. The observable variables (e.g. neural activity) are also very noisy. A typical Kalman filter for motor decoding assumes no control variable and the system can be formulated as two linear equation [102, 103]):

$$\begin{array}{@{}rcl@{}} \vec{x}_{t} &=&A\vec{x}_{t-1} + \vec{w}_{t-1} \\ \vec{y}_{t} &=& C\vec{x}_{t} + \vec{v}_{t} \end{array} $$

where x is the state of the system one want to decode, e.g. joint kinematics or cursor position. y is the observed variables, e.g. neural firing rate. \(\vec {w}_{t}\) and \(\vec {v}_{t}\) are the process and measurement noises drawn from wtN(0,Q) and vtN(0,R) respectively. A, C, Q and R are the Kalman constants that need to be defined according to the decoding model. For the internal state x, if it is a cursor position, it can be expressed as

$$x_{t} = [pos_{t}, vel_{t}, 1]^{T} $$

With the model defined, the Kalman gain K and the estimation error covariance P then can be updated with the typical two-step update equations:

Predict:

$$\begin{array}{@{}rcl@{}} \hat{x}^{-}_{t} &=& A \hat{x}_{t-1} + B u_{t} \\ P^{-}_{t} &=& A P_{t-1} A^{T} +Q \end{array} $$

Update:

$$\begin{array}{@{}rcl@{}} K_{t} &=& \frac{P^{-}_{t}C^{T}}{CP_{t}^{-}C^{T}+R} \\ \hat{x}_{t} &=& \hat{x}^{-}_{t} + K_{t}(y_{t}-C\hat{x}_{t}^{-}) \\ P_{t} &=& (I-K_{t}C)P_{t}^{-} \end{array} $$

where \(\hat {x}^{-}\) and \(\hat {x}\) are the a prior and a posterior state estimates respectively. u is the control variable. Typically it is set to 0 in motor decoding, here we have included it for completeness.

One crucial aspect of performing online motor decoding is the training and re-calibration of the decoding model. Although the neural features for similar movements are relatively stable within a few days [104], the neural tuning curve may start to change when the subject is learning to perform a new task [105]. It is also very difficult to track the same neuron for an extended period of time [106, 107], due to the micro-movement of electrodes and fluctuations of other noise sources. Furthermore, training data are often acquired in an open-loop fashion, meaning that no feedback is provided by the decoder during training. However, in actual decoding session, feedback is provided and the subject may attempt to change his motor imagery in order to “learn” the decoder. This may lead to changes in the underlying neural features [108]. Therefore, re-calibration of the trained model is often necessary and will be ideal if it can be performed online. A successful re-calibration method is the ReFIT-KF algorithm proposed by Gilja et al [109]. ReFIT-KF assumes the subject’s true intention is to move towards the target, so it can generate a pseudo-ground truth from the decoded result automatically even though the prediction of the current model may be wrong. It can then calibrate the model using the estimate ground truth to adapt for the instability of the neural signals. It is able to produce better results than Kalman filter alone [92,93,109].

Due to the more robust signals obtained by intracortical recordings, it has been utilized successfully to help tetraplegia patient control the environments in various ways, including 2D cursor control [73,76,94], virtual and real prosthetic hands [77,79,92,110,111] and functional electrical stimulation of the patients’ own paralyzed hands [90,91,93].

Peripheral decoding of limb movements

Signals from the central nervous system (CNS) eventually arrive at the peripheral nervous system (PNS) and drive the contraction of different muscle fibers. Compared to CNS, signals in the peripheral structures are usually more specific. They contain detailed instructions on the contractions of individual muscle fibers, therefore potentially can enable dexterous prosthetic control. Surgeries involved in peripheral interface are usually less complicated than those involving the intracortical structures. Therefore, many studies are also devoted to motor decoding in the peripheral structures.

Peripheral nerve recordings

Peripheral nerves contain the low-level neural signals sent to activate the contraction of specific muscles. Previous studies on peripheral neural recording mainly focus on afferent sensory information because it is not easy to get efferent signals in anesthetized animals [112]. However, in recent years, more studies have appeared trying to explore the possibility of decoding efferent peripheral nerve signals for prosthetic control. Because the peripheral nerves contain low-level information targeting each muscle, it may be possible to regain high-dexterity and naturalistic control by exploiting this rich information.

One of the major challenges in peripheral nerve recordings is accessing the axons in the nerves. Axons in spinal nerves are bundled in fascicules and multiple fascicules are grouped together to form a peripheral nerve. Those axons are enclosed in three sheaths of connective tissues – the epineurium that covers the entire nerve, and the perineurium that encloses a fascicle and the endoneurium that holds the neurons and blood vessels together within a fascicle. Due to these multiple layers of lamination around an axon, the amplitude of a peripheral nerve signal is usually very small, can be around 5 – 20 μV [112].

There are multiple electrode configurations designed to get a better signal from the peripheral nerves [113]. The cuff electrode [114], as its name suggests, works like a cuff to wrap around a nerve. Its main advantage is that it causes minimal damage to the neural tissues as it does not require any incision on the nerve itself. However, since it only measures the electrical potential at the surface of a nerve, it can only obtain a grand summation of the neural activity in different fascicles. Another variation of the cuff electrode is the flat interface nerve electrode (FINE) [115]. It works like a clip to apply pressure on the nerve and make it flattened into an oval shape, thus increasing its surface area and reducing the distance from the electrode to the fascicles. There are also other types of electrodes that are implanted into the nerves. They offer higher selectivity due to their direct contact with the fascicles. However, they are also more invasive and may cause more damage to the nerve. The longitudinal intrafascicular electrodes (LIFE) are long, thin wires implanted longitudinally into the nerve fascicles [116]. On the other hand, the transverse intrafascicular multichannel electrodes (TIME) are implanted transversely into the nerves, accessing multiple fascicles at the same time. There is also the Utah Slanted Electrode Array [117], which consists of an array of electrodes with different lengths, such that when the array is inserted into the nerve, the tip of the electrode can get into contact with different fascicles. Recently, there is also development of the regenerative peripheral neural interface (RPNI) [118], which uses a muscle graft to wrap around severed fascicles endings. The nerve endings grow into and innervate with the graft, creating a new interface for acquiring neural signal. Of the different types of electrodes introduced, only the cuff electrode is currently used in commercial FDA-approved systems for vagus nerve stimulation (e.g. VNS Therapy, Cyberonics, USA). Most of the others are still in research or undergoing clinical trials [119].

Studies on the human decoding of peripheral signals are still very limited, partly due to the challenge of acquiring nerve signals with sufficient SNR, and may also due to the cross-talk between neural signals and EMG, as the peripheral nerves are usually located in close proximity with the limb musculature. The majority of existing studies focus on upper limb decoding, as upper-limb amputation tends to have a bigger impact on the everyday life of the patients. Neural recording are performed on the ulnar, medial and/or the radial nerve. Different types of electrodes are used, but the more common ones in human decoding are the Utah slate electrode (e.g. in [120,121]) and the LIFE (e.g. [122124]).

The analysis of peripheral signals commonly involves the detection of action potentials in the nerve. The detection procedures are similar to those used in intracortical studies, but the step of clustering spikes is not usually performed. Due to the low SNR of the peripheral signals, sometimes they need to be first de-noised (e.g. by wavelet [124]) before detection. The firing rate of the action potential can then be fed into a regressor (e.g. in [103,120122]) or a classifier (e.g. in [123,124]) for decoding. The difference in using a regressor or a classifier lies in whether a discrete gesture or a continuous joint trajectory is decoded.

Support-vector machine (SVM) is the most commonly used classifier for peripheral decoding (e.g. in [123,124]). For regressor, simple linear regression or a Kalman filter have been used ([103,120122]). Kalman filter allows the online recursive update of the model in real-time, and is especially helpful when the measurement of the target variable is noisy (as often in the case of motor decoding, since it is not possible to measure the actual movement of the missing limb).

The issue of obtaining ground truth for training the decoder is also very important. While for discrete grasp type classification, it may be sufficient to ask the subject to imagine holding a particular grasp, for position decoding a more precise approach have to be used. One common solution is to show a shadow hand on a screen, and ask the subject to try to follow the movement of the hand, either through a manipulandum controlled by the mirrored movement in the intact hand [121] or through imagined phantom limb movements only.

Currently, the performance of human peripheral nerve decoding is still not very satisfactory, partly due to the difficulty in obtaining clear signal and EMG cross-talk. In discrete grasp classification, a 4-class classification task with 3 grasps (power grip, pinch grip, flexion of little finger) and rest have obtained 85% accuracy [124], but state-of-the-art surface electromyogram (EMG) can already distinguish between 7 gestures [125]. Regression-based decoding enables proportional control of a prosthetic hand, and hence can be more intuitive. Decoding based on Kalman filter is able to classify 13 different movements offline, but only 2 movements can be decoded online successfully due to the cross-talk between different degree-of-freedoms (DoFs) [121].

The peripheral nerves offer a promising target for motor decoding. It is more downstream in the motor control pathway and contains more specific information about muscle activities. This property can be potentially exploited to enable high dexterity control. Access to peripheral nerves is also relatively easier than intracortical structures. However, peripheral recordings are plagued by their low SNRs due to the multiple levels of lamination around an axon. This may be improved by better electrode designs, and ultra-low-noise neural amplifiers that can resolve the small amplitude of the nerve signals (e.g. [126]).

Electromyogram (EMG)

EMG signals are the sum of the electrical activities of the muscle fibers, which are triggered by spike trains, i.e. impulses of activation of the innervating motor neurons. EMG signals can be measured in two ways, either on the surface of the skin above a muscle (surface EMG), or directly inside a muscle fiber using a needle electrode (intramuscular EMG). An example of EMG data in different hand gestures is shown in (Fig. 4).

Fig. 4
figure 4

Examples of EMG signal in different hand gestures. Diagram showing EMG signals from 12 surface electrodes in 3 different hand gestures. The original data are from [217]. a EMG signals from both able-bodied and amputee subjects. The last row shows the hand gestures performed for their respective EMG segments. b Locations of the 12 EMG electrodes

Myoelectric signals have been used as the control source for decades in prostheses, in which muscle signals are recorded and translated into control commands to induce prosthesis motions. Intramuscular EMG signals are believed to be of a higher resolution and less susceptible to cross-talks compared with surface EMG because of its more invasive electrode deployment and direct targeting of specific muscles.

Despite decades of research and development, amputees still do not use state-of-the-art myoelectric prostheses more frequently than the basic, body-powered hooks [127], and an estimate of 40% of upper-limb amputees actually reject using a prosthesis [128]. One primary limitation of clinically available hand-prosthesis is the number of simultaneously and proportionally controllable degrees of freedom (DoFs), which is rarely greater than 2 [129,130] and has focused mostly on wrist DoFs without the hand [131], although functions of hand-movement are more essential for daily living.

Myoelectric control can be categorized into direct control and pattern recognition control. Direct control refers to the type of methods that use the amplitude of two surface EMG inputs from an antagonistic muscle pair to control the two directions (ON and OFF) at a prosthetic DoF. Due to the inadequate remaining musculature, signal crosstalk contamination, and attenuation of deep muscle signals at the skin level, the number of independent myosites in the residual forearm is typically limited to two, only allowing the control of one DoF at a time. As a result of this constraint, patients need to toggle between modes using quick co-contraction at the myosites to sequentially control multiple DoFs. Pattern recognition control relies on machine learning algorithms to train a separate classifier for each DoF. Multiple classifiers have been proposed and evaluated, including quadratic discriminant analysis [132], support vector machine [133], artificial neural network [134], hidden Markov models [135], Gaussian mixture models [136], and more. However, as training of the computational models involves the movement of only 1-DoF, the trained classifiers do not support simultaneous control of multiple DoFs. A more promising approach based on machine learning is adopting a regression-based control scheme (instead of classification) that inherently facilitates continuous control (as opposed to ON and OFF), in which a linear or nonlinear mapping from EMG signal features to the changes of prosthesis DoFs is learned. Commonly used methods for this purpose include artificial neural networks [137], support vector machine [138], and kernel ridge regression [131]. A major shortcoming of regression-based control is the requirement for large amount of training data that include an exhaustive combination of movements of all prosthesis DoFs, which is impractical to be clinically implemented.

One of the fundamental issues with EMG based prosthesis control is the scarcity of independent signals with which to control prosthesis DoFs. EMG signals are inherently heavily correlated and lacks the resolution and the information capacity needed for simultaneous and proportional control of multiple DoFs. A potential solution to this problem is to record motor commands directly from the peripheral nerves, such as ulnar and median nerves that directly innervate all five fingers. However, this comes at the costs of invasive surgical implantation of electrodes and the risks of tissue infection and nerve damage.

There have been works to extract more invariant and independent information from EMG signals without invasive recordings. One major group of the efforts focuses on extracting muscle synergy features from EMG recordings, i.e., the complex muscle activation patterns that are executed by users as high-level control inputs regardless of any neurological origin [139]. Muscle synergies are believed to be capable of describing complex force and motion patterns in reduced dimensions and can be used as a robust representation for decoding outputs consistent with user’s intent. Non-negative matrix factorization (NMF) [140] has been commonly used to extract muscle synergies from multichannel EMG signals for simultaneous and proportional control of multiple DOFs [137,141143]. Another group of works focuses on directly extracting the neural codes of motor neuron activities that govern the muscle movements through the nerve pathway. This normally requires advanced recording setups such as high-density EMG with a sufficient number of recording sites that are closely spaced. A number of algorithms have been proposed to extract the underlying neural information [144,145]. Among them, convolution kernel compensation (CKC) has been most extensively used as a type of multichannel blind source separation method [146149]. Despite the promise of extracting neural contents from high-density EMG signals, the demonstration of utilizing such scheme in online experiments remains difficult. More in-depth investigation and significant efforts are needed to build neural interface and achieve direct neural-based control based on this framework.

Decoding of speech motor activities

Although this review mainly focuses on the decoding of movement in the extremities, recently there are also another line of research in decoding motor speech activities [150,151]. Speech production is a complex process involving multiple areas of the brain and dozens of muscles fibers. The muscle activities need to be highly coordinated to produce different speech sounds (i.e. phonemes) which concatenate together to form intelligible words and sentences.

Multiple brain regions are associated with language production [152], but there are two major areas that have received more attentions in speech decoding. The left ventral premotor cortex has been suggested to represent high-level phonemes in speech [153,154], while the ventral sensorimotor cortex contains rich representations of different speech articulators (e.g. lip, tongue, larynx etc.) [155,156]. Therefore most of the decoding efforts concentrate on these two brain regions.

Historically, various neural signals have been exploited to decode speech. EEG is non-invasive but its low signal-to-noise ratio and EMG contamination from facial muscles make it very difficult to be used for decoding speech [151]. There has been some success in using multielectrode array to decode phenomes from multi-unit activities [157]. However, the cortical representation of speech articulators cover a large area that may not be suitable for the very localized recording region of a multielectrode array [156,158]. Furthermore, speech decoding often require overt speech to serve as the ground truth, and that requires the subjects to be capable of speaking clearly. It is difficult to justify implanting penetrating electrodes in the otherwise healthy eloquent cortex to conduct experiments. Currently, ECoG obtains a greater success in speech decoding due to its high signal quality and less invasive nature. ECoG recordings are also commonly employed during brain resection to avoid damage to the eloquent cortex, so it is well-integrated into existing surgical procedures. Studies using ECoG for speech decoding mainly focus on the high gamma band (70-170Hz), as it has been shown that the high gamma activity correlates strongly with ensemble firing rate [159].

Earlier speech decoding efforts have focused on the direct decoding of simple words or phonemes [150,157,158,160162], but their performance is not very satisfactory. Decoding from a limited dictionary or phoneme set may produce a higher accuracy (e.g. >80% for 10 words [160] or 9 phonemes [157]), but it can only cover a very narrow range of human spoken expressions. Studies trying to decode the full range of English phonemes result in a lower classification accuracy (10-50% [150,155,162]). The low classification accuracy can be partly mitigated by incorporating a pronunciation dictionary and language model (e.g. in [150]), which can limit the output of the decoder to more probable words.

On the other hand, recently attentions have been shifted to focus more on decoding the intermediate representation of speech (e.g. articulator movements) rather than decoding phonemes directly. Part of the shift may be motivated by the growing body of evidence suggesting that the speech motor cortex is able to generate differential activation patterns encoding the kinematics of speech articulators [156,163165]. Advances in deep learning has made the prediction of articulator trajectories from acoustic signal (i.e. acoustic-articulatory inversion) accurate enough to act as the ground-truth for decoding, as the traditional ways of implanting coils or magnets in the mouth via articulography is invasive and not compatible with neural recordings [166]. In one very recent study [167], a deep neural network is used to decode ECoG features to articulator trajectories. The trajectories are then decoded by another neural network to acoustic features (e.g. pitch, mel-frequency cepstral coefficients etc.), which are then converted to audible voice using a voice synthesizer. Even mimed speech can be decoded, although with a lower accuracy. In another study [168], ECoG features are decoded into mel-scaled spectrograms directly using a neural network, then a neural network vocoder is used to construct the spectrogram into audible waveforms. These recent results show great promises in decoding human speech from ECoG signals. A summary of the different methods of motor decoding is shown in Table 1.

Table 1 Comparison of different methods for motor decoding

Challenges and future direction

Although great strides have been made in decoding human motor intention, there are still some significant challenges remain to be solved. One of the biggest challenge preventing the adoption of motor decoding outside the laboratory is the limited longevity of the decoding model. Typically, some calibration session is needed to collect data to train the decoding model, then the model is tested on subsequent sessions on the same or next few days. While it is acceptable in a scientific study due to the limited time and clinical resources available, in actual daily use, the trained model must be able to maintain its performance for an extended period of time.

The limited longevity can be due to several reasons. First is the instability of the electrode interfaces. Micro-movement of the electrodes may cause a shift in the feature space. If the decoder is not robust enough, this shift may result in a deterioration of the decoding performance. Another reason is the different environment noises injected into the acquired signals. Neural signals used for decoding usually have a very small amplitude and thus are susceptible to interference by environment noises. A cell-phone, fluorescence lamp or other electrical appliances all inject various types of noise in the acquired signal. As the subjects are performing various tasks in daily lives, they may come into the influence of different noise sources not covered in the trained data set and results in performance degradation. The third reason is the slow build up of immune response on the electrode interface. Glial scars may encapsulate the electrode and increase its impedance [174]. Neurodegeneration as a result of immune response will also lead to a weaker signal [175]. The model longevity problem is multifaceted and must be carefully addressed. First, a better electrode design can help secure the electrode onto its anchoring structure and reduce their relative movement. An implantable solution will also produce more stable feature than one that requires repeated dismantling and reinstallation every time (e.g. EEG and EMG). Second, the model should be trained with more robust features and tested in an environment typical of its everyday use. A shielded chamber may help acquire very clean signals that are good for the demonstration of a prototype. However, it is unlikely that the same quality of signals can be acquired in everyday environment. Thus it is also important to consider how a decoder is tested rather than just looking at offline numerical metrics. Thirdly, advancement in the electrode materials or special organic coatings can potentially reduce its immune response [176]. A flexible instead of rigid electrode may also cause less neuronal damage and inflammation [177,178].

The second challenge is how to account for the difference in features during open-loop training and close-loop control. The training dataset is typically obtained in an open-loop fashion, meaning that the subjects are instructed to carry out a particular motor imagery without any feedback. However, in actual use the system will provide feedback to the subject based on the decoder outputs. When the decoder output is wrong, the subject may try to correct it deliberately, and that may lead to discrepancy in the offline and online performance [179]. One of the solutions is to introduce a small calibration session with feedback at the beginning of the testing session, like in many EEG-based motor decoding studies. The original model is trained with an open-loop paradigm, then the model is further fine-tuned with feedback in the calibration session. However, this is only possible if a clear ground truth is available. For the case in which the ground truth is not available, e.g. in the case of a tetraplegic patient where it is very difficult to know the true intention of the subject, the ReFIT algorithm is another approach to address this problem [109]. The basic idea of the ReFIT algorithm is that it tries to construct a pseudo ground truth by assuming that the subject is constantly trying to correct the wrong output of the decoder. Thus the directional vector of the motor intention is taken to be always pointing towards the target from the current cursor position. Using this method, it is possible to train a decoder from scratch with as few as 3 min of data [94]. Online calibration with feedback can offer a more realistic prediction on how the decoder is able to perform in real-life. This approach can also let the decoder quickly adapt to any shift in the feature space due to change in the electrode interface or environmental noises. However, online calibration demands that the model can be updated quickly, which puts an constraint on the complexity of the decoding model. More research is needed to study how to update the decoder efficiently in real-time.

Besides advancement in decoding algorithms, development of new electrodes and neural amplifiers also play a very important part in advancing motor decoding. Recent trends in electrode development mainly focus on improving four areas of electrode design: density, flexibility, biocompatibility and connectivity. Denser electrode can improve the spatial resolution of neural recordings. High-density electrode has been created from silicon wafer and carbon fiber monofilament [180,181]. Electrode material with a flexibility closer to that of brain tissues can reduce neural damage and inflammatory response. Many flexible polymers have been used to make neural electrode, including polyimide [182,183], parylene [184], PDMS [185] etc. Biocompatibility is always an important issue in electrode design because inflammatory response and encapsulation deteriorate signal quality over time and undermine the quality of chronic neural recordings. Strategies to improve biocompatibility including using inert metals like gold or platinum, using flexible materials to reduce tissue damage, or coating the electrode with biocompatible materials like conducting polymer [186] and carbon nanotubes [187]. Read-out connection from the electrodes will also quickly become a problem when the density and number of electrode continue to increase. Incorporating transistors into the electrodes directly to enable connection multiplexing is one of the ways to mitigate this problem [188,189]. Readers interested in neural electrode designs are suggested to consult other more in-depth reviews in this area [119,172,176,177,190].

Development of neural amplifiers also plays a very important role in advancing the science of motor decoding, as we first need to acquire a clear neural signal before any processing and decoding can be done. There are multiple lines of research trying to improve the different aspects of the amplifier design. Firstly, the power consumption of an amplifier can be reduced by resource sharing (e.g. one amplifier sharing multiple electrodes [191] or multiple amplifiers sharing one analog-to-digital convertor [192]), power scheduling (e.g. switching off unused components [193], dynamically adjusting the amplifier parameters [194]), or reduction of supply voltage [195]. Secondly, the channel count can be increased by multiplexing or integrating amplifiers directly with the electrodes [191,196]. Thirdly, the circuit noise can be reduced by trimming [197], chopping [198,199], auto-zeroing [200] or frequency-shaping [201] etc.. Fourthly, wireless transmission of power or data can be achieved by an inductive link [193,202,203], short-distance power harvest [193,204] or even ultrasound [205]. Finally, the functionality of the amplifier can also be expanded by integrating more signal processing on-chip, e.g. spike detection [203], spike sorting [206,207] and data compression [208,209]. Interested readers are encouraged to consult other more focused reviews in this area [210213].

Conclusions

Every year, a large number of patients suffer from various degrees of movement disability due to amputation or neurological disorders. Their everyday lives and works are severely affected. With modern neurotechnology, it is now possible to intercept and decode the motor intention at different points along the neuro-muscular control pathway and use that information to drive a prosthetic device to restore movement. In this paper, we have reviewed the various signal features and techniques to decode motor intention in human. Although motor decoding performance is improving steadily with the advancements in electrode configurations, neural amplifier designs and decoding algorithms, we are still very far away from the goal of achieving naturalistic and dexterous control like our native limbs. The eventual successful clinical application of motor decoding will depend on the concerted efforts of both healthcare and engineering professionals, and likely also needs to be tailored-made according to the conditions and ability of each patient. We hope our review can provide a useful overview of the current state-of-the-art in motor decoding, so that researchers interested in the field can be aware of the neural features that they can exploit, potential problems they may encounter and the available solutions that they can adopt.

Availability of data and material

All data analysed during this study are included in their corresponding original articles [214217].

Abbreviations

AAR:

adaptive auto-regression

CAR:

common average reference

CKC:

convolution kernel compensation CSP: common spatial pattern

CNS:

central nervous system

DoF:

degree of freedom

ECoG:

electrocorticogram

EEG:

electroencephalography

EMG:

electromyogram

ERD:

event-related desynchronization

ERS:

event-related synchronization

FINE:

flat interface nerve electrode

GPi:

internal globus pallidus

LIFE:

logitudinal intrafascicular electrodes

LMP:

local motor potential

M1:

primary motor cortex

NMF:

non-negative matrix factorization

NHP:

non-human primate

PMA:

premotor area

PNS:

peripheral nervous system

PPC:

posterior parietal cortex

RPNI:

regenerative peripheral neural interface

SMA:

supplementary motor area

SMR:

sensorimotor rhythm

SNR:

signal-to-noise ratio

STN:

subthalamus nucleus

TIME:

transverse intrafascicular multichannel electrodes

VLo:

oral part of ventral lateral nucleus

References

  1. Ziegler-Graham K, MacKenzie EJ, Ephraim PL, Travison TG, Brookmeyer R. Estimating the Prevalence of Limb Loss in the United States: 2005 to 2050. Arch Phys Med Rehabil. 2008; 89(3):422–9. https://doi.org/10.1016/j.apmr.2007.11.005. http://arxiv.org/abs/arXiv:0803.1592v1.

    Google Scholar 

  2. Adams PF, Hendershot GE, Marano MA. Current estimates from the National Health Interview Survey. Vital Health Stat. 1999; 10(200):1996. https://doi.org/10.1037/1099-9809.6.2.168.

    Google Scholar 

  3. Wing-Kin T, Kai-yu T, Fei M, Shangkai Gao. A Minimal Set of Electrodes for Motor Imagery BCI to Control an Assistive Device in Chronic Stroke Subjects: A Multi-Session Study. IEEE Tran Neural Syst Rehabil Eng. 2011; 19(6):617–27. https://doi.org/10.1109/TNSRE.2011.2168542.

    Google Scholar 

  4. Ang KK, Chua KSG, Phua KS, Wang C, Chin ZY, Kuah CWK, Low W, Guan C. A Randomized Controlled Trial of EEG-Based Motor Imagery Brain-Computer Interface Robotic Rehabilitation for Stroke. Clin EEG Neurosci. 2015; 46(4):310–20. https://doi.org/10.1177/1550059414522229.

    Google Scholar 

  5. Bear MF, Connors BW, Paradiso MA. Neuroscience: Exploring the Brain, 4th editio. Philadelphia: Wolters Kluwer Health; 2015.

    Google Scholar 

  6. Kandel ER, Schwartz JH, Jessell TM. Principles of Neural Science, 4th editio. New York: McGraw-Hill Medical; 2000.

    Google Scholar 

  7. Andersen RA, Buneo CA. Intentional maps in posterior parietal cortex,. Ann Rev Neurosci. 2002; 25:189–220. https://doi.org/10.1146/annurev.neuro.25.112701.142922.

    Google Scholar 

  8. Weinrich M, Wise SP. The premotor cortex of the monkey,. J Neurosci. 1982; 2(9):1329–45.

    Google Scholar 

  9. Calabresi P, Picconi B, Tozzi A, Ghiglieri V, Di Filippo M. Direct and indirect pathways of basal ganglia: A critical reappraisal. Nat Neurosci. 2014; 17(8):1022–30. https://doi.org/10.1038/nn.3743.

    Google Scholar 

  10. DeLong MR. Primate models of movement disorders of basal ganglia origin. Trends Neurosci. 1990; 13(7):281–5. https://doi.org/10.1016/0166-2236(90)90110-V.

    Google Scholar 

  11. Albin RL, Young AB, Penney JB. The functional anatomy of basal ganglia disorders,. Trends Neurosci. 1989; 12(10):366–75. https://doi.org/10.1093/jhered/esy024.

    Google Scholar 

  12. Graziano M. The Organization of Behavioral Repertoire in Motor Cortex. Ann Rev Neurosci. 2006; 29(1):105–134. https://doi.org/10.1146/annurev.neuro.29.051605.112924.

    Google Scholar 

  13. Splittgerber R. Snell’s Clinical Neuroanatomy, 8th. Philadelphia: Wolters Kluwer; 2019.

    Google Scholar 

  14. Buzsáki G, Anastassiou Ca, Koch C. The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes. Nat Rev Neurosci. 2012; 13(6):407–20. https://doi.org/10.1038/nrn3241. NIHMS150003.

    Google Scholar 

  15. Webster JG. Medical Instrumentation Application and Design, 4th. New Jersey: Wiley and Sons; 2009.

    Google Scholar 

  16. Chatrian GE, Petersen MC, Lazarte JA. The blocking of the rolandic wicket rhythm and some central changes related to movement. Electroencephalogr Clin Neurophysiol. 1959; 11(3):497–510. https://doi.org/10.1016/0013-4694(59)90048-3.

    Google Scholar 

  17. Schomer DL, da Silva FL. Niedermeyer’s Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Philadelphia: Lippincott Williams & Wilkins; 2011.

    Google Scholar 

  18. McFarland DJJ, Miner LAa, Vaughan TMM, Wolpaw JRR. Mu and beta rhythm topographies during motor imagery and actual movements,. Brain Topogr. 2000; 12(3):177–86.

    Google Scholar 

  19. Han Y, Bin H. Brain-Computer Interfaces Using Sensorimotor Rhythms: Current State and Future Perspectives. IEEE Trans Biomed Eng. 2014; 61(5):1425–35. https://doi.org/10.1109/TBME.2014.2312397. NIHMS150003.

    Google Scholar 

  20. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clin Neuropsychol. 2002; 113(6):767–91.

    Google Scholar 

  21. Graimann B, Huggins JE, Levine SP, Pfurtscheller G. Visualization of significant ERD/ERS patterns in multichannel EEG and ECoG data. Clin Neuropsychol. 2002; 113(1):43–7. https://doi.org/10.1016/S1388-2457(01)00697-6.

    Google Scholar 

  22. Pfurtscheller G, Lopes FH. Event-related EEG / MEG synchronization and desynchronization : basic principles. Clinical Neurophysiology. 1999; 110:1842–57. http://dx.doi.org/10.1016/S1388-2457(99)00141-8. http://arxiv.org/abs/S1388-2457(99)00141-8..

    Google Scholar 

  23. Pfurtscheller G, Stancák a, Edlinger G. On the existence of different types of central beta rhythms below 30 Hz,. Electroencephalogr Clin Neurophysiol. 1997; 102(4):316–25.

    Google Scholar 

  24. Chen R, Yaseen Z, Cohen LG, Hallett M. Time course of corticospinal excitability in reaction time and self-paced movements. Ann Neurol. 1998; 44(3):317–25. https://doi.org/10.1002/ana.410440306. Accessed 5 Jan 2019.

    Google Scholar 

  25. Pfurtscheller G, Neuper C, Andrew C, Edlinger G. Foot and hand area mu rhythms. Int J Psychophysiol. 1997; 26:121–35.

    Google Scholar 

  26. Blankertz B, Losch F, Krauledat M, Dornhege G, Curio G, Müller K-R. The Berlin Brain–Computer Interface: accurate performance from first-session in BCI-naïve subjects,. IEEE Trans Biomed Eng. 2008; 55(10):2452–62. https://doi.org/10.1109/TBME.2008.923152.

    Google Scholar 

  27. Feng C, Wang H, Lu N, Chen T, He H, Lu Y, Tu XM. Log-transformation and its implications for data analysis,. Shanghai Arch Psychiatry. 2014; 26(2):105–9. https://doi.org/10.3969/j.issn.1002-0829.2014.02.009.

    Google Scholar 

  28. Schlögl A, Flotzinger D, Pfurtscheller G. Adaptive autoregressive modeling used for single-trial EEG classification. Biomed Tech. 1997; 42(6):162–7. https://doi.org/10.1515/bmte.1997.42.6.162.

    Google Scholar 

  29. Pfurtscheller G, Neuper C, Schlögl a, Lugger K. Separability of EEG signals recorded during right and left motor imagery using adaptive autoregressive parameters. IEEE Trans Rehabil Eng Publ IEEE Eng Med Biol Soc. 1998; 6(3):316–25.

    Google Scholar 

  30. Schalk G, Kubánek J, Miller KJ, Anderson NR, Leuthardt EC, Ojemann JG, Limbrick D, Gerhardt LA, Moran D, Wolpaw JR. Decoding two-dimensional movement trajectories using electrocorticographic signals in humans,. J Neural Eng. 2007; 4(3):264–75. https://doi.org/10.1088/1741-2560/4/3/012.

    Google Scholar 

  31. McFarland DJ, Wolpaw JR. Sensorimotor rhythm-based brain-computer interface (BCI): Model order selection for autoregressive spectral analysis. J Neural Eng. 2008; 5(2):155–62. https://doi.org/10.1088/1741-2560/5/2/006.

    Google Scholar 

  32. Haykin S. Adaptive filter theory, (5th ed). New Jersey: Prentice-Hall; 2014.

    MATH  Google Scholar 

  33. Wang T, Deng J, He B. Classifying EEG-based motor imagery tasks by means of time-frequency synthesized spatial patterns. Clin Neuropsychol. 2004; 115(12):2744–53. https://doi.org/10.1016/j.clinph.2004.06.022.

    Google Scholar 

  34. Yamawaki N, Wilke C, Liu Z, He B. An enhanced time-frequency-spatial approach for motor imagery classification,. IEEE Trans Neural Syst Rehabil Eng Publ IEEE Eng Med Biol Soc. 2006; 14(2):250–4. https://doi.org/10.1109/TNSRE.2006.875567. NIHMS150003.

    Google Scholar 

  35. Mcfarland DJ, Mccane LM, David SV, Wolpaw JR. Spatial filter selection for EEG-based communication. Electroencephalogr Clin Neurophysiol. 1997; 103:386–94.

    Google Scholar 

  36. Ramoser H, Müller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng. 2000; 8(4):441–6. https://doi.org/10.1109/86.895946.

    Google Scholar 

  37. Guger C, Ramoser H, Pfurtscheller G. Real-time EEG analysis with subject-specific spatial patterns for a brain-computer interface (BCI),. IEEE Trans Rehabil Eng. 2000; 8(4):447–56.

    Google Scholar 

  38. Ang KK, Chin ZY, Zhang H, Guan C. Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), 2390–2397: 2008. https://doi.org/10.1109/IJCNN.2008.4634130.

  39. Townsend G, Graimann B, Pfurtscheller G. Continuous EEG classification during motor imagery–simulation of an asynchronous BCI,. IEEE Trans Neural Syst Rehabil Eng. 2004; 12(2):258–65. https://doi.org/10.1109/TNSRE.2004.827220.

    Google Scholar 

  40. Wolpaw JR, McFarland DJ. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans,. Proc Natl Acad Sci U S A. 2004; 101(51):17849–54. https://doi.org/10.1073/pnas.0403504101.

    Google Scholar 

  41. Royer AS, Doud AJ, Rose ML, He B. EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies,. IEEE Trans Rehabil Eng Publ IEEE Eng Med Biol Soc. 2010; 18(6):581–9. https://doi.org/10.1109/TNSRE.2010.2077654.

    Google Scholar 

  42. McFarland DJ, Sarnacki WA, Wolpaw JR. Electroencephalographic (EEG) control of three-dimensional movement. J Neural Eng. 2010; 7(3). http://dx.doi.org/10.1088/1741-2560/7/3/036007. 036007.

  43. Lafleur K, Cassady K, Doud A, Shades K, Rogin E, He B. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain-computer interface. J Neural Eng. 2013; 10(4). http://dx.doi.org/10.1088/1741-2560/10/4/046003. NIHMS150003.

  44. Shiman F, López-Larraz E, Sarasola-Sanz A, Irastorza-Landa N, Spüler M, Birbaumer N, Ramos-Murguialday A. Classification of different reaching movements from the same limb using EEG. J Neural Eng. 2017; 14(4). https://doi.org/10.1088/1741-2552/aa70d2.

  45. Yong X, Menon C. EEG classification of different imaginary movements within the same limb. PLoS ONE. 2015; 10(4):1–24. https://doi.org/10.1371/journal.pone.0121896.

    Google Scholar 

  46. Liao K, Xiao R, Gonzalez J, Ding L. Decoding individual finger movements from one hand using human EEG signals. PLoS ONE. 2014; 9(1):1–12. https://doi.org/10.1371/journal.pone.0085192.

    Google Scholar 

  47. Fei Meng, Kai-yu Tong, Suk-tak Chan, Wan-wa Wong, Ka-him Lui, Kwok-wing Tang, Xiaorong Gao, Shangkai Gao. BCI-FES training system design and implementation for rehabilitation of stroke patients. In: 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). Hong Kong: IEEE: 2008. p. 4103–6. https://doi.org/10.1109/IJCNN.2008.4634388. http://ieeexplore.ieee.org/document/4634388/.

    Google Scholar 

  48. Tam W-K, Ke Z, Tong K-Y. Performance of common spatial pattern under a smaller set of EEG electrodes in brain-computer interface on chronic stroke patients: A multi-session dataset study Conference proceedings : … Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Eng Med Biol Soc Conf. 2011; 2011:6344–7. https://doi.org/10.1109/IEMBS.2011.6091566.

    Google Scholar 

  49. Aminoff MJ. Aminoff’s Electrodiagnosis in Clinical Neurology. Philadelphia: Elsevier; 2012. https://doi.org/10.1016/C2010-0-65599-4. Accessed 5 Jan 2019.

    Google Scholar 

  50. Jiang T, Liu S, Pellizzer G, Aydoseli A, Karamursel S, Sabanci PA, Sencer A, Gurses C, Ince NF. Characterization of Hand Clenching in Human Sensorimotor Cortex Using High-, and Ultra-High Frequency Band Modulations of Electrocorticogram. Front Neurosci. 2018;12(February). https://doi.org/10.3389/fnins.2018.00110.

  51. Schalk G, Miller KJ, Anderson NR, Wilson JA, Smyth MD, Ojemann JG, Moran DW, Wolpaw JR, Leuthardt EC. Two-dimensional movement control using electrocorticographic signals in humans. J Neural Eng. 2008; 5(1):75–84. https://doi.org/10.1088/1741-2560/5/1/008.

    Google Scholar 

  52. Kramer MA, Kirsch HE, Szeri AJ. Pathological pattern formation and cortical propagation of epileptic seizures. J R Soc Interface. 2005; 2(2):113–27. https://doi.org/10.1098/rsif.2004.0028.

    Google Scholar 

  53. Wang Z, Gunduz A, Brunner P, Ritaccio AL, Ji Q, Schalk G. Decoding onset and direction of movements using Electrocorticographic (ECoG) signals in humans. Front Neuroengineering. 2012; 5(August):1–13. https://doi.org/10.3389/fneng.2012.00015.

    Google Scholar 

  54. Hotson G, McMullen DP, Fifer MS, Johannes MS, Katyal KD, Para MP, Armiger R, Anderson WS, Thakor NV, Wester BA, Crone NE. Individual finger control of a modular prosthetic limb using high-density electrocorticography in a human subject. J Neural Eng. 2016; 13(2). http://dx.doi.org/10.1088/1741-2560/13/2/026017. 15334406.

  55. Branco MP, Freudenburg ZV, Aarnoutse EJ, Bleichner MG, Vansteensel MJ, Ramsey NF. Decoding hand gestures from primary somatosensory cortex using high-density ECoG. NeuroImage. 2017; 147(December 2016):130–42. https://doi.org/10.1016/j.neuroimage.2016.12.004.

    Google Scholar 

  56. Bleichner MG, Freudenburg ZV, Jansma JM, Aarnoutse EJ, Vansteensel MJ, Ramsey NF. Give me a sign: decoding four complex hand gestures based on high-density ECoG. Brain Struct Funct. 2016; 221(1):203–16. https://doi.org/10.1007/s00429-014-0902-x.

    Google Scholar 

  57. Spüler M, Walter A, Ramos-Murguialday A, Naros G, Birbaumer N, Gharabaghi A, Rosenstiel W, Bogdan M. Decoding of motor intentions from epidural ECoG recordings in severely paralyzed chronic stroke patients. J Neural Eng. 2014; 11(6). https://doi.org/10.1088/1741-2560/11/6/066008.

  58. Pistohl T, Ball T, Schulze-Bonhage A, Aertsen A, Mehring C. Prediction of arm movement trajectories from ECoG-recordings in humans,. J Neurosci Methods. 2008; 167(1):105–14. https://doi.org/10.1016/j.jneumeth.2007.10.001.

    Google Scholar 

  59. Kubánek J, Miller KJ, Ojemann JG, Wolpaw JR, Schalk G. Decoding flexion of individual fingers using electrocorticographic signals in humans. J Neural Eng. 2009; 6(6). http://dx.doi.org/10.1088/1741-2560/6/6/066001. NIHMS150003.

  60. Milekovic T, Fischer J, Pistohl T, Ruescher J, Schulze-Bonhage A, Aertsen A, Rickert J, Ball T, Mehring C. An online brain-machine interface using decoding of movement direction from the human electrocorticogram. J Neural Eng. 2012; 9(4). https://doi.org/10.1088/1741-2560/9/4/046003.

  61. Flamary R, Rakotomamonjy A. Decoding finger movements from ECoG signals using switching linear models. Front Neurosci. 2012; 6(MAR):1–9. https://doi.org/10.3389/fnins.2012.00029. http://arxiv.org/abs/1106.3395.

    Google Scholar 

  62. Hammer J, Fischer J, Ruescher J, Schulze-Bonhage A, Aertsen A, Ball T. The role of ECoG magnitude and phase in decoding position, velocity, and acceleration during continuous motor behavior. Frontiers in Neuroscience. 2013; 7(7 NOV):1–15. https://doi.org/10.3389/fnins.2013.00200.

    Google Scholar 

  63. Liang N, Bougrain L. Decoding finger flexion from band-specific ecog signals in humans. Front Neurosci. 2012; 6(JUN):1–6. https://doi.org/10.3389/fnins.2012.00091. http://arxiv.org/abs/0907.5500.

    Google Scholar 

  64. Xie Z, Schwartz O, Prasad A. Decoding of finger trajectory from ECoG using deep learning. J Neural Eng. 2018; 15(3). https://doi.org/10.1088/1741-2552/aa9dbe.

  65. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW. A brain-computer interface using electrocorticographic signals in humans. J Neural Eng. 2004; 1(2):63–71. http://dx.doi.org/10.1088/1741-2560/1/2/001. http://arxiv.org/abs/1741-2560/1/2/001.

    Google Scholar 

  66. Leuthardt EC, Miller KJ, Schalk G, Rao RPN, Ojemann JG. Electrocorticography-based brain computer interface - The seattle experience. IEEE Trans Neural Syst Rehabil Eng. 2006; 14(2):194–8. https://doi.org/10.1109/TNSRE.2006.875536.

    Google Scholar 

  67. Pistohl T, Schulze-Bonhage A, Aertsen A, Mehring C, Ball T. Decoding natural grasp types from human ECoG. NeuroImage. 2012; 59(1):248–60. https://doi.org/10.1016/j.neuroimage.2011.06.084.

    Google Scholar 

  68. Acharya S, Fifer MS, Benz HL, Crone NE, Thakor NV. Electrocorticographic amplitude predicts finger positions during slow grasping motions of the hand. J Neural Eng. 2010; 7(4). http://dx.doi.org/10.1088/1741-2560/7/4/046002. http://arxiv.org/abs/arXiv:1011.1669v3.

  69. Gomez-Rodriguez M, Grosse-Wentrup M, Peters J, Naros G, Hill J, Schölkopf B, Gharabaghi A. Epidural ECoG online decoding of arm movement intention in hemiparesis Proceedings - Workshop on Brain Decoding: Pattern Recognition Challenges in Neuroimaging, WBD 2010. In: Conjunction with theInternational Conference on Pattern Recognition, ICPR 2010, 36–39: 2010. https://doi.org/10.1109/WBD.2010.17.

  70. Yanagisawa T, Hirata M, Saitoh Y, Kishima H, Matsushita K, Goto T, Fukuma R, Yokoi H, Kamitani Y, Yoshimine T. Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann Neurol. 2012; 71(3):353–61. https://doi.org/10.1002/ana.22613.

    Google Scholar 

  71. Georgopoulos AP, Kalaska JF, Caminiti R, Massey JT. On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. J Neurosci. 1982; 2(11):1527–37. https://doi.org/10.1080/10705422.2016.1269379.

    Google Scholar 

  72. Yousry TA, Schmid UD, Alkadhi H, Schmidt D, Peraud A, Buettner A, Winkler P. Localization of the motor hand area to a knob on the precentral gyrus. A new landmark. Brain. 1997; 120:141–57.

    Google Scholar 

  73. Hochberg LR, Serruya MD, Friehs GM, Mukand Ja, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature. 2006; 442(7099):164–71. https://doi.org/10.1038/nature04970.

    Google Scholar 

  74. Truccolo W, Friehs GM, Donoghue JP, Hochberg LR. Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia,. J Neurosci Off J Soc Neurosci. 2008; 28(5):1163–78. https://doi.org/10.1523/JNEUROSCI.4415-07.2008.

    Google Scholar 

  75. Kim SP, Simeral JD, Hochberg LR, Donoghue JP, Black MJ. Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J Neural Eng. 2008; 5(4):455–76. https://doi.org/10.1088/1741-2560/5/4/010.

    Google Scholar 

  76. Simeral JD, Kim SP, Black MJ, Donoghue JP, Hochberg LR. Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J Neural Eng. 2011; 8(2). http://dx.doi.org/10.1088/1741-2560/8/2/025027. NIHMS150003.

  77. Chadwick EK, Blana D, Simeral JD, Lambrecht J, Kim SP, Cornwell AS, Taylor DM, Hochberg LR, Donoghue JP, Kirsch RF. Continuous neuronal ensemble control of simulated arm reaching by a human with tetraplegia. J Neural Eng. 2011; 8(3). http://dx.doi.org/10.1088/1741-2560/8/3/034003. NIHMS150003.

  78. Andersen RA, Buneo CA. Intentional Maps in Posterior Parietal Cortex. Ann Rev Neurosci. 2002; 25(1):189–220. https://doi.org/10.1146/annurev.neuro.25.112701.142922.

    Google Scholar 

  79. Aflalo T, Kellis S, Klaes C, Lee B, Shi Y, Pejsa K, Shanfield K, Hayes-Jackson S, Aisen M, Heck C, Liu C, Andersen R. Decoding motor imagery from the posterior parietal cortex of a tetraplecig human. Science. 2015; 348(6237):906–10. http://dx.doi.org/10.7910/DVN/GJDUTV. 15334406.

    Google Scholar 

  80. Lee C, Rohrer WH, Sparks DL. Population coding of saccadic eye movements by neurons in the superior colliculus. Nature. 1988; 332(6162):357–60. https://doi.org/10.1038/332357a0.

    Google Scholar 

  81. Gibson S, Judy JW, Marković D. Technology-aware algorithm design for neural spike detection, feature extraction, and dimensionality reduction. IEEE Trans Neural Syst Rehabil Eng Publ IEEE Eng Med Biol Soc. 2010; 18(5):469–78. https://doi.org/10.1109/TNSRE.2010.2051683.

    Google Scholar 

  82. Tam W-k, So R., Guan C, Yang Z. EC-PC spike detection for high performance brain-computer interface. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Milan: IEEE: 2015. p. 5142–5145. https://doi.org/10.1109/EMBC.2015.7319549. http://ieeexplore.ieee.org/document/7319549/.

    Google Scholar 

  83. Mukhopadhyay S, Ray GC. A new interpretation of nonlinear energy operator and its efficacy in spike detection,. IEEE Trans Biomed Eng. 1998; 45(2):180–7. https://doi.org/10.1109/10.661266.

    Google Scholar 

  84. Nenadic Z, Burdick JW. Spike detection using the continuous wavelet transform,. IEEE Trans Biomed Eng. 2005; 52(1):74–87. https://doi.org/10.1109/TBME.2004.839800.

    Google Scholar 

  85. Ge D, Farina D. Spike Sorting. In: Introduction to Neural Engineering for Motor Rehabilitation. Hoboken, NJ, USA: John Wiley & Sons, Inc.: 2013. p. 155–172. https://doi.org/10.1002/9781118628522.ch8. http://doi.wiley.com/10.1002/9781118628522.ch8.

    Google Scholar 

  86. Lefebvre B, Yger P, Marre O. Recent progress in multi-electrode spike sorting methods. J Physiol Paris. 2016. https://doi.org/10.1016/j.jphysparis.2017.02.005.

  87. Lewicki MS. A review of methods for spike sorting: the detection and classification of neural action potentials,. Netw Comput Neural Sys. 1998; 9(4):53–78.

    MathSciNet  MATH  Google Scholar 

  88. Gilja V, Pandarinath C, Blabe CH, Nuyujukian P, Simeral JD, Sarma AA, Sorice BL, Perge JA, Jarosiewicz B, Hochberg LR, Shenoy KV, Henderson JM. Clinical translation of a high-performance neural prosthesis. Nat Med. 2015; 21(10):1142–5. https://doi.org/10.1038/nm.3953. 15334406.

    Google Scholar 

  89. Sharma G, Friedenberg DA, Annetta N, Glenn B, Bockbrader M, Majstorovic C, Domas S, Mysiw WJ, Rezai A, Bouton C. Using an artificial neural bypass to restore cortical control of rhythmic movements in a human with quadriplegia. Sci Rep. 2016; 6(August):1–11. https://doi.org/10.1038/srep33807.

    Google Scholar 

  90. Bouton CE, Shaikhouni A, Annetta NV, Bockbrader MA, Friedenberg DA, Nielson DM, Sharma G, Sederberg PB, Glenn BC, Mysiw WJ, Morgan AG, Deogaonkar M, Rezai AR. Restoring cortical control of functional movement in a human with quadriplegia. Nature. 2016; 533(7602):247–50. https://doi.org/10.1038/nature17435.

    Google Scholar 

  91. Friedenberg DA, Schwemmer MA, Landgraf AJ, Annetta NV, Bockbrader MA, Bouton CE, Zhang M, Rezai AR, Mysiw WJ, Bresler HS, Sharma G. Neuroprosthetic-enabled control of graded arm muscle contraction in a paralyzed human. Sci Rep. 2017; 7(1):1–10. https://doi.org/10.1038/s41598-017-08120-9.

    Google Scholar 

  92. Pandarinath C, Nuyujukian P, Blabe CH, Sorice BL, Saab J, Willett FR, Hochberg LR, Shenoy KV, Henderson JM. High performance communication by people with paralysis using an intracortical brain-computer interface. eLife. 2017; 6:1–27. https://doi.org/10.7554/eLife.18554.

    Google Scholar 

  93. Ajiboye AB, Willett FR, Young DR, Memberg WD, Murphy BA, Miller JP, Walter BL, Sweet JA, Hoyen HA, Keith MW, Peckham PH, Simeral JD, Donoghue JP, Hochberg LR, Kirsch RF. Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration. Lancet. 2017; 389(10081):1821–30. http://dx.doi.org/10.1016/S0140-6736(17)30601-3. 15334406.

    Google Scholar 

  94. Brandman DM, Hosman T, Saab J, Burkhart MC, Shanahan BE, Ciancibello JG, Sarma AA, Milstein DJ, Vargas-Irwin CE, Franco B, Kelemen J, Blabe C, Murphy BA, Young DR, Willett FR, Pandarinath C, Stavisky SD, Kirsch RF, Walter BL, Bolu Ajiboye A, Cash SS, Eskandar EN, Miller JP, Sweet JA, Shenoy KV, Henderson JM, Jarosiewicz B, Harrison MT, Simeral JD, Hochberg LR. Rapid calibration of an intracortical brain-computer interface for people with tetraplegia. J Neural Eng. 2018; 15(2). https://doi.org/10.1088/1741-2552/aa9ee7.

  95. Georgopoulos AP, Schwartz AB, Kettner RE. Neuronal population coding of movement direction. Science. 1986; 233(4771):1416–9. https://doi.org/10.1126/science.3749885. 0111115.

    Google Scholar 

  96. Georgopoulos aP, Kettner RE, Schwartz aB. Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population,. J Neurosci Off J Soc Neurosci. 1988; 8(8):2928–37.

    Google Scholar 

  97. Amirikian B, Georgopulos AP. Directional tuning profiles of motor cortical cells. Neurosci Res. 2000; 36(1):73–9. https://doi.org/10.1016/S0168-0102(99)00112-1.

    Google Scholar 

  98. Salinas E, Abbott LF. Vector reconstruction from firing rates,. J Comput Neurosci. 1994; 1(1-2):89–107.

    MATH  Google Scholar 

  99. Wu W, Gao Y, Bienenstock E, Donoghue JP, Black MJ. Bayesian population decoding of motor cortical activity using a Kalman filter,. Neural Comput. 2006; 18(1):80–118. https://doi.org/10.1162/089976606774841585.

    MathSciNet  MATH  Google Scholar 

  100. Kim SP, Simeral JD, Hochberg LR, Donoghue JP, Friehs GM, Black MJ. Point-and-click cursor control with an intracortical neural interface system by humans with tetraplegia. IEEE Trans Neural Syst Rehabil Eng. 2011; 19(2):193–203. https://doi.org/10.1109/TNSRE.2011.2107750.

    Google Scholar 

  101. Bishop G, Welch G. An introduction to the Kalman filter. In: Proc of SIGGRAPH, Course 8: 2001. p. 27599–3175. doi:. http://arxiv.org/abs/1011.1669.

  102. Orsborn AL, Moorman HG, Overduin SA, Shanechi MM, Dimitrov DF, Carmena JM. Closed-loop decoder adaptation shapes neural plasticity for skillful neuroprosthetic control. Neuron. 2014; 82(6):1380–93. https://doi.org/10.1016/j.neuron.2014.04.048.

    Google Scholar 

  103. Vu PP, Irwin ZT, Bullard AJ, Ambani SW, Sando IC, Urbanchek MG, Cederna PS, Chestek CA. Closed-Loop Continuous Hand Control via Chronic Recording of Regenerative Peripheral Nerve Interfaces. IEEE Trans Neural Syst Rehabil Eng. 2018; 26(2):515–26. https://doi.org/10.1109/TNSRE.2017.2772961.

    Google Scholar 

  104. Chestek CA, Batista AP, Santhanam G, Yu BM, Afshar A, Cunningham JP, Gilja V, Ryu SI, Churchland MM, Shenoy KV. Single-Neuron Stability during Repeated Reaching in Macaque Premotor Cortex. J Neurosci. 2007; 27(40):10742–50. https://doi.org/10.1523/JNEUROSCI.0959-07.2007.

    Google Scholar 

  105. Rokni U, Richardson AG, Bizzi E, Seung HS. Motor Learning with Unstable Neural Representations. Neuron. 2007; 54(4):653–66. https://doi.org/10.1016/j.neuron.2007.04.030.

    Google Scholar 

  106. Dickey A. S., Suminski A., Amit Y., Hatsopoulos N. G.Single-Unit Stability Using Chronically Implanted Multielectrode Arrays. J Neurophysiol. 2009; 102(2):1331–9. https://doi.org/10.1152/jn.90920.2008.

    Google Scholar 

  107. Fraser GW, Schwartz AB. Recording from the same neurons chronically in motor cortex. J Neurophysiol. 2012; 107(7):1970–8. https://doi.org/10.1152/jn.01012.2010. NIHMS150003.

    Google Scholar 

  108. Jarosiewicz B, Masse NY, Bacher D, Cash SS, Eskandar E, Friehs G, Donoghue JP, Hochberg LR. Advantages of closed-loop calibration in intracortical brain-computer interfaces for people with tetraplegia. J Neural Eng. 2013; 10(4):046012. https://doi.org/10.1088/1741-2560/10/4/046012.

    Google Scholar 

  109. Gilja V, Nuyujukian P, Chestek Ca, Cunningham JP, Yu BM, Fan JM, Churchland MM, Kaufman MT, Kao JC, Ryu SI, Shenoy KV. A high-performance neural prosthesis enabled by control algorithm design,. Nat Neurosci. 2012; 15(12):1752–7. https://doi.org/10.1038/nn.3265.

    Google Scholar 

  110. Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, Haddadin S, Liu J, Cash SS, van der Smagt P., Donoghue J. P.Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature. 2012; 485(7398):372–5. https://doi.org/10.1038/nature11076.

    Google Scholar 

  111. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, McMorland AJC, Velliste M, Boninger ML, Schwartz AB. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet. 2013; 381(9866):557–64,. https://doi.org/10.1016/S0140-6736(12)61816-9.

    Google Scholar 

  112. Micera S, Carpaneto J, Raspopovic S. Control of hand prostheses using peripheral information. IEEE Rev Biomed Eng. 2010; 3:48–68. https://doi.org/10.1109/RBME.2010.2085429.

    Google Scholar 

  113. del Valle J, Navarro X. Interfaces with the Peripheral Nerve for the Control of Neuroprostheses. Int Rev Neurobiol. 2013; 109:63–83. https://doi.org/10.1016/B978-0-12-420045-6.00002-X.

    Google Scholar 

  114. Hoffer JA, Loeb GE. Implantable electrical and mechanical interfaces with nerve and muscle. Ann Biomed Eng. 1980; 8(4-6):351–60. https://doi.org/10.1007/BF02363438.

    Google Scholar 

  115. Leventhal DK, Durand DM. Subfascicle Stimulation Selectivity with the Flat Interface Nerve Electrode. Ann Biomed Eng. 2003; 31(6):643–52. https://doi.org/10.1114/1.1569266.

    Google Scholar 

  116. Yoshida K, Stein RB. Characterization of signals and noise rejection with bipolar longitudinal intrafascicular electrodes. IEEE Trans Biomed Eng. 1999; 46(2):226–34. https://doi.org/10.1109/10.740885.

    Google Scholar 

  117. Boretius T, Badia J, Pascual-Font A, Schuettler M, Navarro X, Yoshida K, Stieglitz T. A transverse intrafascicular multichannel electrode (TIME) to interface with the peripheral nerve. Biosens Bioelectron. 2010; 26(1):62–9. https://doi.org/10.1016/j.bios.2010.05.010.

    Google Scholar 

  118. Irwin ZT, Schroeder KE, Vu PP, Tat DM, Bullard AJ, Woo SL, Sando IC, Urbanchek MG, Cederna PS, Chestek CA. Chronic recording of hand prosthesis control signals via a regenerative peripheral nerve interface in a rhesus macaque. J Neural Eng. 2016; 13(4). https://doi.org/10.1088/1741-2560/13/4/046007.

  119. Russell C, Roche AD, Chakrabarty S. Peripheral nerve bionic interface: a review of electrodes. Int J Intell Robot Appl. 2019; 3(1):11–8. https://doi.org/10.1007/s41315-019-00086-3.

    Google Scholar 

  120. Warren DJ, Kellis S, Nieveen JG, Wendelken SM, Dantas H, Davis TS, Hutchinson DT, Normann RA, Clark GA, Mathews VJ. Recording and decoding for neural prostheses. Proceedings of the IEEE. 2016; 104(2):374–91. https://doi.org/10.1109/JPROC.2015.2507180.

    Google Scholar 

  121. Davis TS, Wark HAC, Hutchinson DT, Warren DJ, O’Neill K, Scheinblum T, Clark GA, Normann RA, Greger BAO, Greger BO. Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves. J Neural Eng. 2016; 13(3). http://orcid.org/0000-0002-6702-7596.

  122. Dhillon GS, Lawrence SM, Hutchinson DT, Horch KW. Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs. J Hand Surg. 2004; 29(4):605–15. https://doi.org/10.1016/j.jhsa.2004.02.006.

    Google Scholar 

  123. Micera S, Rigosa J, Carpaneto J, Citi L, Raspopovic S, Guglielmelli E, Benvenuto A, Rossini L, Di Pino G, Cavallo G, Carrozza MC, Cipriani C, Hoffmann KP, Dario P, Rossini PM. On the control of a robot hand by extracting neural signals from the PNS: Preliminary results from a human implantation; 2009. https://doi.org/10.1109/IEMBS.2009.5332764.

  124. Micera S, Rossini PM, Rigosa J, Citi L, Carpaneto J, Raspopovic S, Tombini M, Cipriani C, Assenza G, Carrozza MC, Hoffmann KP, Yoshida K, Navarro X, Dario P. Decoding of grasping information from neural signals recorded using peripheral intrafascicular interfaces. J NeuroEngineering Rehabil. 2011; 8(1):2–11. https://doi.org/10.1186/1743-0003-8-53.

    Google Scholar 

  125. Cipriani C, Antfolk C, Controzzi M, Lundborg G, Rosen B, Carrozza MC, Sebelius F. Online myoelectric control of a dexterous hand prosthesis by transradial amputees. IEEE Trans Neural Syst Rehabil Eng. 2011; 19(3):260–70. https://doi.org/10.1109/TNSRE.2011.2108667.

    Google Scholar 

  126. Yang Z, Xu J, Nguyen AT, Wu T, Zhao W, Tam W-k. Neuronix enables continuous, simultaneous neural recording and electrical microstimulation. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando: IEEE: 2016. p. 4451–4454. https://doi.org/10.1109/EMBC.2016.7591715. http://ieeexplore.ieee.org/document/7591715/.

    Google Scholar 

  127. Biddiss E, Chau T. Upper limb prosthesis use and abandonment: A survey of the last 25 years. Prosthet Orthot Int. 2007; 31(3):236–57. https://doi.org/10.1080/03093640600994581.

    Google Scholar 

  128. Raichle KA, Hanley MA, Molton I, Kadel NJ, Campbell K, Phelps E, Ehde D, Smith DG. Prosthesis use in persons with lower- and upper-limb amputation,. J Rehabil Res Dev. 2008; 45(7):961–72. https://doi.org/10.1109/JMEMS.2005.859083.A. NIHMS150003.

    Google Scholar 

  129. Young AJ, Smith LH, Rouse EJ, Hargrove LJ. A comparison of the real-time controllability of pattern recognition to conventional myoelectric control for discrete and simultaneous movements. J NeuroEngineering Rehabil. 2014; 11(1):1–10. https://doi.org/10.1186/1743-0003-11-5.

    Google Scholar 

  130. Jiang N, Rehbaum H, Vujaklija I, Graimann B, Farina D. Intuitive, online, simultaneous, and proportional myoelectric control over two degrees-of-freedom in upper limb amputees. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2014; 22(3):501–10. https://doi.org/10.1109/TNSRE.2013.2278411.

    Google Scholar 

  131. Hahne JM, Bießmann F, Jiang N, Rehbaum H, Farina D, Meinecke FC, Muller KR, Parra LC. Linear and nonlinear regression techniques for simultaneous and proportional myoelectric control. IEEE Trans Neural Syst Rehabil Eng. 2014; 22(2):269–79. https://doi.org/10.1109/TNSRE.2014.2305520.

    Google Scholar 

  132. Kim KS, Choi HH, Moon CS, Mun CW. Comparison of k-nearest neighbor, quadratic discriminant and linear discriminant analysis in classification of electromyogram signals based on the wrist-motion directions. Curr Appl Phys. 2011; 11(3):740–5. https://doi.org/10.1016/j.cap.2010.11.051.

    Google Scholar 

  133. Oskoei MA, Hu H. Support vector machine-based classification scheme for myoelectric control applied to upper limb. IEEE Trans Biomed Eng. 2008; 55(8):1956–65. https://doi.org/10.1109/TBME.2008.919734.

    Google Scholar 

  134. Hargrove LJ, Englehart K, Hudgins B. A comparison of surface and intramuscular myoelectric signal classification. IEEE Trans Biomed Eng. 2007; 54(5):847–53. https://doi.org/10.1109/TBME.2006.889192.

    Google Scholar 

  135. Chan ADC, Englehart KB. Continuous myoelectric control for powered prostheses using hidden Markov models,. IEEE Trans Biomed Eng. 2005; 52(1):121–4. https://doi.org/10.1109/TBME.2004.836492.

    Google Scholar 

  136. Huang Y, Englehart KB, Hudgins B, Chan ADC. A Gaussian Mixture Model Based Classification Scheme for Myoelectric Control of Powered Upper Limb Prostheses. IEEE Trans Biomed Eng. 2005; 52(11):1801–11. https://doi.org/10.1109/TBME.2005.856295.

    Google Scholar 

  137. Jiang N, Vujaklija I, Rehbaum H, Graimann B, Farina D. Is accurate mapping of EMG signals on kinematics needed for precise online myoelectric control?,. IEEE Trans Neural Syst Rehabil Eng Publ IEEE Eng Med Biol Soc. 2014; 22(3):549–58. https://doi.org/10.1109/TNSRE.2013.2287383.

    Google Scholar 

  138. Ameri A, Kamavuako EN, Scheme EJ, Englehart KB, Parker PA. Support vector regression for improved real-time, simultaneous myoelectric control. IEEE Trans Neural Syst Rehabil Eng. 2014; 22(6):1198–209. https://doi.org/10.1109/TNSRE.2014.2323576.

    Google Scholar 

  139. Ison M, Artemiadis P. The role of muscle synergies in myoelectric control: Trends and challenges for simultaneous multifunction control. J Neural Eng. 2014; 11(5). https://doi.org/10.1088/1741-2560/11/5/051001.

  140. Lee DD, Seung HS. Algorithms for non-negative matrix factorization. Adv Neural Inf Proces Syst. 2001; 13(NIPS 2000):556–62.

    Google Scholar 

  141. Muceli S, Jiang N, Farina D. Extracting signals robust to electrode number and shift for online simultaneous and proportional myoelectric control by factorization algorithms. IEEE Trans Neural Syst Rehabil Eng. 2014; 22(3):623–33. https://doi.org/10.1109/TNSRE.2013.2282898.

    Google Scholar 

  142. Jiang N, Englehart K, Parker P. Extracting Simultaneous and Proportional Neural Control Information of Multiple Degree of Freedom from the Surface Electromyographic Signal. IEEE Trans Biomed Eng. 2009; 56(4):1070–80.

    Google Scholar 

  143. Rehbaum H, Jiang N, Paredes L, Amsuess S, Graimann B, Farina D. Real time simultaneous and proportional control of multiple degrees of freedom from surface EMG: Preliminary results on subjects with limb deficiency. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 1346–1349: 2012. https://doi.org/10.1109/EMBC.2012.6346187.

  144. De Luca CJ, Adam A, Wotiz R, Gilmore LD, Nawab SH. Decomposition of Surface EMG Signals. J Neurophysiol. 2006; 96(3):1646–57. https://doi.org/10.1152/jn.00009.2006.

    Google Scholar 

  145. Gazzoni M, Farina D, Merletti R. A new method for the extraction and classification of single motor unit action potentials from surface EMG signals. J Neurosci Methods. 2004; 136(2):165–77. https://doi.org/10.1016/j.jneumeth.2004.01.002.

    Google Scholar 

  146. Holobar A, Farina D, Gazzoni M, Merletti R, Zazula D. Estimating motor unit discharge patterns from high-density surface electromyogram. Clin Neuropsychol. 2009; 120(3):551–62. https://doi.org/10.1016/j.clinph.2008.10.160.

    Google Scholar 

  147. Holobar A, Minetto MA, Botter A, Negro F, Farina D. Experimental Analysis of Accuracy in the Identification of Motor Unit Spike Trains. IEEE Trans Neural Syst Rehabil Eng. 2010; 18(3):221–9.

    Google Scholar 

  148. Holobar A, Glaser V, Gallego JA, Dideriksen JL, Farina D. Non-invasive characterization of motor unit behaviour in pathological tremor. J Neural Eng. 2012; 9(5). https://doi.org/10.1088/1741-2560/9/5/056011.

  149. Glaser V, Holobar A, Zazula D. Real-time motor unit identification from high-density surface EMG. IEEE Trans Neural Syst Rehabil Eng. 2013; 21(6):949–58. https://doi.org/10.1109/TNSRE.2013.2247631.

    Google Scholar 

  150. Herff C, Heger D, de Pesters A, Telaar D, Brunner P, Schalk G, Schultz T. Brain-to-text: Decoding spoken phrases from phone representations in the brain. Front Neurosci. 2015; 9(MAY):1–11. https://doi.org/10.3389/fnins.2015.00217.

    Google Scholar 

  151. Herff C, Schultz T. Automatic speech recognition from neural signals: A focused review. Front Neurosci. 2016; 10(SEP):1–7. https://doi.org/10.3389/fnins.2016.00429.

    Google Scholar 

  152. Hickok G. Computational neuroanatomy of speech production. Nat Rev Neurosci. 2012; 13(2):135–45. https://doi.org/10.1038/nrn3158.

    Google Scholar 

  153. Siok WT, Jin Z, Fletcher P, Tan LH. Distinct brain regions associated with syllable and phoneme. Hum Brain Mapp. 2003; 18(3):201–7. https://doi.org/10.1002/hbm.10094.

    Google Scholar 

  154. Guenther F. H., Ghosh S. S., Tourville J. A.Neural modeling and imaging of the cortical interactions underlying syllable production. Brain and Language. 2006; 96(3):280–301. https://doi.org/10.1016/j.bandl.2005.06.001.

    Google Scholar 

  155. Mugler EM, Tate MC, Livescu K, Templer JW, Goldrick MA, Slutzky MW. Differential Representation of Articulatory Gestures and Phonemes in Precentral and Inferior Frontal Gyri. J Neurosci. 2018; 38(46):9803–13. https://doi.org/10.1523/jneurosci.1206-18.2018.

    Google Scholar 

  156. Bouchard KE, Mesgarani N, Johnson K, Chang EF. Functional organization of human sensorimotor cortex for speech articulation. Nature. 2013; 495(7441):327–32. https://doi.org/10.1038/nature11911.

    Google Scholar 

  157. Stavisky SD, Rezaii P, Willett FR, Hochberg LR, Shenoy KV, Henderson JM. Decoding Speech from Intracortical Multielectrode Arrays in Dorsal ’Arm/Hand Areas’ of Human Motor Cortex. Proc Ann Int Conf IEEE Eng Med Biol Soc EMBS. 2018; 2018-July:93–7. https://doi.org/10.1109/EMBC.2018.8512199.

    Google Scholar 

  158. Bouchard KE, Chang EF. Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography; 2014. https://doi.org/10.1109/EMBC.2014.6945185.

  159. Ray S, Crone NE, Niebur E, Franaszczuk PJ, Hsiao SS. J Neurosci Off J Soc Neurosci. 2008; 28(45):11526–36. https://doi.org/10.1523/JNEUROSCI.2848-08.2008.

  160. Kellis S, Miller K, Thomson K, Brown R, House P, Greger B. Decoding spoken words using local field potentials recorded from the cortical surface. J Neural Eng. 2010; 7(5). https://doi.org/10.1088/1741-2560/7/5/056007.

  161. Blakely T, Miller KJ, Rao RPN, Holmes MD, Ojemann JG. Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids. 2009:4964–4967. https://doi.org/10.1109/iembs.2008.4650328.

  162. Mugler EM, Patton JL, Flint RD, Wright ZA, Schuele SU, Rosenow J, Shih JJ, Krusienski DJ, Slutzky MW. Direct classification of all American English phonemes using signals from functional speech motor cortex. J Neural Eng. 2014; 11(3). https://doi.org/10.1088/1741-2560/11/3/035015.

  163. Lotte F, Brumberg JS, Brunner P, Gunduz A, Ritaccio AL, Guan C, Schalk G. Electrocorticographic representations of segmental features in continuous speech. Front Hum Neurosci. 2015; 09(February):1–13. https://doi.org/10.3389/fnhum.2015.00097.

    Google Scholar 

  164. Carey D, Krishnan S, Callaghan MF, Sereno MI, Dick F. Functional and Quantitative MRI Mapping of Somatomotor Representations of Human Supralaryngeal Vocal Tract. Cereb Cortex. 2017; 27(1):265–78. https://doi.org/10.1093/cercor/bhw393.

    Google Scholar 

  165. Chartier J, Anumanchipalli GK, Johnson K, Chang EF. Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor Cortex. Neuron. 2018; 98(5):1042–10544. https://doi.org/10.1016/j.neuron.2018.04.031.

    Google Scholar 

  166. Schultz T, Wand M, Hueber T, Krusienski DJ, Herff C, Brumberg JS. Biosignal-Based Spoken Communication: A Survey. IEEE/ACM Trans Audio Speech Lang Process. 2017; 25(12):2257–71. https://doi.org/10.1109/TASLP.2017.2752365.

    Google Scholar 

  167. Anumanchipalli GK, Chartier J, Chang EF. Speech synthesis from neural decoding of spoken sentences. Nature. 2019; 568(7753):493–8. https://doi.org/10.1038/s41586-019-1119-1.

    Google Scholar 

  168. Angrick M, Herff C, Mugler E, Tate MC, Slutzky MW, Krusienski DJ, Schultz T. Speech synthesis from ECoG using densely connected 3D convolutional neural networks. J Neural Eng. 2019; 16(3):036019. https://doi.org/10.1088/1741-2552/ab0c59.

    Google Scholar 

  169. Burle B, Spieser L, Roger C, Casini L, Hasbroucq T, Vidal F. Spatial and temporal resolutions of EEG: Is it really black and white? A scalp current density view. Int J Psychophysiol. 2015; 97(3):210–20. https://doi.org/10.1016/j.ijpsycho.2015.05.004.

    Google Scholar 

  170. Ward MP, Rajdev P, Ellison C, Irazoqui PP. Toward a comparison of microelectrodes for acute and chronic recordings. Brain Res. 2009; 1282:183–200. https://doi.org/10.1016/j.brainres.2009.05.052.

    Google Scholar 

  171. Malagodi MS, Horch KW, Schoenberg AA. An intrafascicular electrode for recording of action potentials in peripheral nerves. Ann Biomed Eng. 1989; 17(4):397–410. https://doi.org/10.1007/BF02368058.

    Google Scholar 

  172. Rijnbeek EH, Eleveld N, Olthuis W. Update on peripheral nerve electrodes for closed-loop neuroprosthetics. Front Neurosci. 2018; 12(MAY):1–9. https://doi.org/10.3389/fnins.2018.00350.

    Google Scholar 

  173. Rau G, Disselhorst-Klug C. Principles of high-spatial-resolution surface EMG (HSR-EMG): Single motor unit detection and application in the diagnosis of neuromuscular disorders. J Electromyogr Kinesiol. 1997; 7(4):233–9. https://doi.org/10.1016/S1050-6411(97)00007-2.

    Google Scholar 

  174. Polikov VS, Tresco Pa, Reichert WM. Response of brain tissue to chronically implanted neural electrodes. J Neurosc Methods. 2005; 148:1–18. https://doi.org/10.1016/j.jneumeth.2005.08.015.

    Google Scholar 

  175. McConnell GC, Rees HD, Levey AI, Gutekunst CA, Gross RE, Bellamkonda RV. Implanted neural electrodes cause chronic, local inflammation that is correlated with local neurodegeneration. J Neural Eng. 2009; 6(5). https://doi.org/10.1111/cup.12876.

  176. Aregueta-Robles UA, Woolley AJ, Poole-Warren LA, Lovell NH, Green RA. Organic electrode coatings for next-generation neural interfaces. Front Neuroengineering. 2014; 7(May):1–18. https://doi.org/10.3389/fneng.2014.00015.

    Google Scholar 

  177. Fattahi P, Yang G, Kim G, Abidian MR. A review of organic and inorganic biomaterials for neural interfaces. Adv Mater. 2014; 26(12):1846–85. https://doi.org/10.1002/adma.201304496. 15334406.

    Google Scholar 

  178. Lago N, Yoshida K, Koch KP, Navarro X. Assessment of biocompatibility of chronically implanted polyimide and platinum intrafascicular electrodes. IEEE Trans Biomed Eng. 2007; 54(2):281–90. https://doi.org/10.1109/TBME.2006.886617.

    Google Scholar 

  179. Chase SM, Schwartz AB, Kass RE. Neural Netw Off J Int Neural Netw Soc. 2009; 22(9):1203–13. https://doi.org/10.1016/j.neunet.2009.05.005.

  180. Santacruz SR, Hou JF, Carmena JM, Maharbiz MM, Pister KSJ, Massey TL. A high-density carbon fiber neural recording array technology. J Neural Eng. 2018; 16(1):016024. https://doi.org/10.1088/1741-2552/aae8d9.

    Google Scholar 

  181. Wark HAC, Sharma R, Mathews KS, Fernandez E, Yoo J, Christensen B, Tresco P, Rieth L, Solzbacher F, Normann RA, Tathireddy P. A new high-density (25 electrodes/mm2) penetrating microelectrode array for recording and stimulating sub-millimeter neuroanatomical structures. J Neural Eng. 2013; 10(4). https://doi.org/10.1088/1741-2560/10/4/045003.

  182. Cheung KC, Renaud P, Tanila H, Djupsund K. Flexible polyimide microelectrode array for in vivo recordings and current source density analysis. Biosens Bioelectron. 2007; 22(8):1783–90. https://doi.org/10.1016/j.bios.2006.08.035.

    Google Scholar 

  183. Rubehn B, Bosman C, Oostenveld R, Fries P, Stieglitz T. A MEMS-based flexible multichannel ECoG-electrode array. J Neural Eng. 2009; 6(3). https://doi.org/10.1088/1741-2560/6/3/036003.

  184. Pellinen DS, Moon T, Vetter RJ, Miriani R, Kipke DR. Multifunctional Flexible Parylene-Based Intracortical Microelectrodes. 2006:5272–5275. https://doi.org/10.1109/iembs.2005.1615669.

  185. Guo L, Meacham KW, Hochman S, DeWeerth SP. A PDMS-based conical-well microelectrode array for surface stimulation and recording of neural tissues. IEEE Transactions on Biomedical Engineering. 2010; 57(10 PART 1):2485–94. https://doi.org/10.1109/TBME.2010.2052617.

    Google Scholar 

  186. Wilks SJ, Richardson-burns SM, Hendricks JL, Martin DC, Otto KJ. Poly (3, 4-ethylenedioxythiophene) as a micro-neural interface material for electrostimulation. Front J Women Stud. 2009; 2(June):1–8. https://doi.org/10.3389/neuro.16.007.

    Google Scholar 

  187. Keefer EW, Botterman BR, Romero MI, Rossi AF, Gross GW. Carbon nanotube coating improves neuronal recordings. Nat Nanotechnol. 2008; 3(7):434–9. https://doi.org/10.1038/nnano.2008.174.

    Google Scholar 

  188. Viventi J, Kim DH, Vigeland L, Frechette ES, Blanco JA, Kim YS, Avrin AE, Tiruvadi VR, Hwang SW, Vanleer AC, Wulsin DF, Davis K, Gelber CE, Palmer L, Van Der Spiegel J, Wu J, Xiao J, Huang Y, Contreras D, Rogers JA, Litt B. Flexible, foldable, actively multiplexed, high-density electrode array for mapping brain activity in vivo. Nat Neurosci. 2011; 14(12):1599–605. https://doi.org/10.1038/nn.2973.

    Google Scholar 

  189. Escabí MA, Read HL, Viventi J, Kim D-H, Higgins NC, Storace DA, Liu ASK, Gifford AM, Burke JF, Campisi M, Kim Y-S, Avrin AE, der Spiegel Jan V, Huang Y, Li M, Wu J, Rogers JA, Litt B, Cohen YE. A high-density, high-channel count, multiplexed μECoG array for auditory-cortex recordings. J Neurophysiol. 2014; 112(6):1566–83. https://doi.org/10.1152/jn.00179.2013.

    Google Scholar 

  190. Fekete Z. Recent advances in silicon-based neural microelectrodes and microsystems: A review. Sensors Actuators B Chem. 2015; 215:300–15. https://doi.org/10.1016/j.snb.2015.03.055.

    Google Scholar 

  191. Lopez CM, Andrei A, Mitra S, Welkenhuysen M, Eberle W, Bartic C, Puers R, Yazicioglu RF, Gielen GGE. An Implantable 455-Active-Electrode 52-Channel CMOS Neural Probe. IEEE J Solid-State Circ. 2014; 49(1):248–61. https://doi.org/10.1109/JSSC.2013.2284347.

    Google Scholar 

  192. Moo SC, Zhi Y, Yuce MR, Linh H, Liu W. A 128-Channel 6 mW Wireless Neural Recording IC With Spike Feature Extraction and UWB Transmitter. IEEE Trans Neural Syst Rehabil Eng. 2009; 17(4):312–21. https://doi.org/10.1109/TNSRE.2009.2021607.

    Google Scholar 

  193. Lee SB, Lee HM, Kiani M, Jow UM, Ghovanloo M. An inductively powered scalable 32-channel wireless neural recording system-on-a-chip for neuroscience applications. IEEE Trans Biomed Cir Syst. 2010; 4(6 PART 1):360–71. https://doi.org/10.1109/TBCAS.2010.2078814.

    Google Scholar 

  194. Wattanapanitch W, Sarpeshkar R. A Low-Power 32-Channel Digitally Programmable Neural Recording Integrated Circuit. IEEE Trans Biomed Cir Syst. 2011; 5(6):592–602. https://doi.org/10.1109/TBCAS.2011.2163404.

    Google Scholar 

  195. Dong Han, Yuanjin Zheng, Rajkumar R., Dawe G. S., Minkyu Je. A 0.45 V 100-Channel Neural-Recording IC With Sub-uW/Channel Consumption in 0.18 um CMOS. IEEE Trans Biomed Cir Syst. 2013; 7(6):735–46. https://doi.org/10.1109/TBCAS.2014.2298860.

    Google Scholar 

  196. Jun JJ, Steinmetz NA, Siegle JH, Denman DJ, Bauza M, Barbarits B, Lee AK, Anastassiou CA, Andrei A, Aydin C, Barbic M, Blanche TJ, Bonin V, Couto J, Dutta B, Gratiy SL, Gutnisky DA, Hausser M, Karsh B, Ledochowitsch P, Lopez CM, Mitelut C, Musa S, Okun M, Pachitariu M, Putzeys J, Rich PD, Rossant C, Sun W, Svoboda K, Carandini M, Harris KD, Koch C, O’Keefe J, Harris TD. Fully Integrated Silicon Probes for High-Density Recording of Neural Activity. Nature. 2017; in press(7679):232–6. https://doi.org/10.1038/nature24636.

    Google Scholar 

  197. Bagheri A, Salam MT, Velazquez JLP, Genov R. Low-Frequency Noise and Offset Rejection in DC-Coupled Neural Amplifiers: A Review and Digitally-Assisted Design Tutorial. IEEE Trans Biomed Cir Syst. 2017; 11(1):161–76. https://doi.org/10.1109/TBCAS.2016.2539518.

    Google Scholar 

  198. Denison T, Consoer K, Santa W, Avestruz AT, Cooley J, Kelly A. A2 μw 100 nV/rtHz Chopper-Stabilized Instrumentation Amplifier for Chronic Measurement of Neural Field Potentials. IEEE J Solid-State Circ. 2007; 42(12):2934–45. https://doi.org/10.1109/JSSC.2007.908664.

    Google Scholar 

  199. Fan Q, Sebastiano F, Huijsing JH, Makinwa KAA. A 1.8 μ W 60 nV/ rtHz capacitively-coupled chopper instrumentation amplifier in 65 nm CMOS for wireless sensor nodes. IEEE J Solid-State Circ. 2011; 46(7):1534–43. https://doi.org/10.1109/JSSC.2011.2143610.

    Google Scholar 

  200. Chan CH, Wills J, LaCoss J, Granacki JJ, Choma J. A micro-power low-noise auto-zeroing CMOS amplifier for cortical neural prostheses. In: IEEE 2006 Biomedical Circuits and Systems Conference Healthcare Technology, BioCAS 2006, 214–217: 2006. https://doi.org/10.1109/BIOCAS.2006.4600346.

  201. Xu J, Wu T, Liu W, Yang Z. A Frequency Shaping Neural Recorder With 3 pF Input Capacitance and 11 Plus 4.5 Bits Dynamic Range. IEEE Trans Biomed Cir Syst. 2014; 8(4):510–27. https://doi.org/10.1109/TBCAS.2013.2293821.

    Google Scholar 

  202. Borton DA, Yin M, Aceros J, Nurmikko A. An implantable wireless neural interface for recording cortical circuit dynamics in moving primates. J Neural Eng. 2013; 10(2). https://doi.org/10.1088/1741-2560/10/2/026010.

  203. Harrison R, Watkins P, Kier R, Lovejoy R, Black D, Normann R, Solzbacher F. A Low-Power Integrated Circuit for a Wireless 100-Electrode Neural Recording System. In: 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers. San Francisco: IEEE: 2006. p. 2258–2267. https://doi.org/10.1109/ISSCC.2006.1696288. http://ieeexplore.ieee.org/document/1696288/.

    Google Scholar 

  204. Jow UM, McMenamin P, Kiani M, Manns JR, Ghovanloo M. EnerCage: A smart experimental arena with scalable architecture for behavioral experiments. IEEE Trans Biomed Eng. 2014; 61(1):139–48. https://doi.org/10.1109/TBME.2013.2278180.

    Google Scholar 

  205. Seo D, Carmena JM, Rabaey JM, Alon E, Maharbiz MM. Neural Dust: An Ultrasonic, Low Power Solution for Chronic Brain-Machine Interfaces. 2013; April. http://arxiv.org/abs/1307.2196.

  206. Horiuchi T, Swindell T, Sander D, Abshier P. A low-power CMOS neural amplifier with amplitude measurements for spike sorting. IEEE. https://doi.org/10.1109/ISCAS.2004.1328932. http://ieeexplore.ieee.org/document/1328932/.

  207. Chae M, Liu W, Yang Z, Chen T, Kim J, Sivaprakasam M, Yuce M. A 128-channel 6mW wireless neural recording IC with on-the-fly spike sorting and UWB Tansmitter. In: Digest of Technical Papers - IEEE International Solid-State Circuits Conference: 2008. p. 146–8. https://doi.org/10.1109/ISSCC.2008.4523099.

  208. Liu X., Zhang M., Xiong T., Richardson A. G., Lucas T. H., Chin P. S., Etienne-Cummings R., Tran T. D., Van der Spiegel J.A Fully Integrated Wireless Compressed Sensing Neural Signal Acquisition System for Chronic Recording and Brain Machine Interface. IEEE Trans Biomed Cir Syst. 2016; 10(4):874–83. https://doi.org/10.1109/TBCAS.2016.2574362.

    Google Scholar 

  209. Aziz JNY, Abdelhalim K, Shulyzki R, Genov R, Bardakjian BL, Derchansky M, Serletis D, Carlen PL. 256-channel neural recording and delta compression microsystem with 3D electrodes. IEEE J Solid State Circuits. 2009; 44(3):995–1005. https://doi.org/10.1109/JSSC.2008.2010997.

    Google Scholar 

  210. Ruther P., Paul O.New approaches for CMOS-based devices for large-scale neural recording. Curr Opin Neurobiol. 2015; 32:31–37. https://doi.org/10.1016/j.conb.2014.10.007.

    Google Scholar 

  211. Liu X, Van der Spiegel J. Neural Recording Front-End Design. In: Brain-Machine Interface. Cham: Springer: 2018. p. 17–68. http://link.springer.com/10.1007/978-3-319-67940-2_2.

    Google Scholar 

  212. Bharucha E, Sepehrian H, Gosselin B. A Survey of Neural Front End Amplifiers and Their Requirements toward Practical Neural Interfaces. J Low Power Electron Appl. 2014; 4(4):268–91. https://doi.org/10.3390/jlpea4040268.

    Google Scholar 

  213. Ng KA, Greenwald E, Xu YP, Thakor NV. Implantable neurotechnologies: a review of integrated circuit neural amplifiers. Med Biol Eng Comput. 2016; 54(1):45–62. https://doi.org/10.1007/s11517-015-1431-3.

    Google Scholar 

  214. Tangermann M, Müller K-R, Aertsen A, Birbaumer N, Braun C, Brunner C, Leeb R, Mehring C, Miller KJ, Müller-Putz GR, Nolte G, Pfurtscheller G, Preissl H, Schalk G, Schlögl A, Vidaurre C, Waldert S, Blankertz B. Review of the BCI Competition IV,. Front Neurosci. 2012; 6(July):55. https://doi.org/10.3389/fnins.2012.00055.

    Google Scholar 

  215. Lawlor PN, Perich MG, Miller LE, Kording KP. Linear-nonlinear-time-warp-poisson models of neural activity. J Comput Neurosci. 2018; 173–191. https://doi.org/10.1007/s10827-018-0696-6.

  216. Perich MG, Lawlor PN, Kording KP, Miller LE. Extracellular neural recordings from macaque primary and dorsal premotor motor cortex during a sequential reaching task. 2018. https://doi.org/10.6080/K0FT8J72.

  217. Krasoulis A, Kyranou I, Erden MS, Nazarpour K, Vijayakumar S. Improved prosthetic hand control with concurrent use of myoelectric and inertial measurements. J NeuroEngineering Rehabil. 2017; 14(1):1–14. https://doi.org/10.1186/s12984-017-0284-4.

    Google Scholar 

Download references

Funding

This work was supported in part by the DARPA under Grants HR0011-17-2-0060 and N66001-15-C-4016, in part by the NIH under Grant R01-MH111413-01, in part by MnDRIVE Program at the University of Minnesota and in part by NSF CAREER Award #1845709.

Author information

Authors and Affiliations

Authors

Contributions

WKT is responsible for the majority of the paper. TW writes the section in EMG motor decoding and helps edit the whole paper. QZ provides guidance on machine learning algorithms to decode neural signals. EK provides guidance on physiology. ZY edits the paper and provides overall guidance for this review.

Corresponding author

Correspondence to Zhi Yang.

Ethics declarations

Ethics approval and consent to participate

This review includes re-analysis of data from other published studies. Each of the cited studies has received ethics approval in their respective institutions.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tam, Wk., Wu, T., Zhao, Q. et al. Human motor decoding from neural signals: a review. BMC biomed eng 1, 22 (2019). https://doi.org/10.1186/s42490-019-0022-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s42490-019-0022-z

Keywords