Elsevier

NeuroImage

Volume 50, Issue 4, 1 May 2010, Pages 1438-1445
NeuroImage

Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI

https://doi.org/10.1016/j.neuroimage.2010.01.062Get rights and content

Abstract

Functional magnetic resonance imaging (fMRI) data and electroencephalography (EEG) data provide complementary spatio-temporal information about brain function. Methods to couple the relative strengths of these modalities usually involve two stages: first forming a feature set from each dataset based on one criterion followed by exploration of connections among the features using a second criterion. We propose a data fusion method for simultaneously acquired fMRI and EEG data that combines these steps using a single criterion for finding the cross-modality associations and performing source separation. Using multi-set canonical correlation analysis (M-CCA), we obtain a decomposition of the two modalities, into spatial maps for fMRI data and a corresponding temporal evolution for EEG data, based on trial-to-trial covariation across the two modalities. Additionally, the analysis is performed on data from a group of subjects in order to make group inferences about the covariation across modalities. Being multivariate, the proposed method facilitates the study of brain connectivity along with localization of brain function. M-CCA can be easily extended to incorporate different data types and additional modalities. We demonstrate the promise of the proposed method in finding covarying trial-to-trial amplitude modulations (AMs) in an auditory task involving implicit pattern learning. The results show approximately linear decreasing trends in AMs for both modalities and the corresponding spatial activations occur mainly in motor, frontal, temporal, inferior parietal, and orbito-frontal areas that are linked both to sensory function as well as learning and expectation—all of which match activations related to the presented paradigm.

Introduction

Brain imaging techniques such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide complementary information about the brain function. Although fMRI provides an indirect hemodynamic measure of brain function that serves as a surrogate for neuronal activity, it offers good spatial localization of blood oxygen dependent activity. On the other hand, while EEG is a more direct measure of the brain's electrical activity measured instantaneously at excellent temporal resolution, it does not provide satisfactory spatial location of the source of the activation. Integrating the information from these two imaging modalities promises to provide a better understanding of brain function, see e.g., (Calhoun and Adalı, 2009, Hopfinger et al., 2005, Horwitz and Poeppel, 2002, Makeig et al., 2002).

A number of methods have been developed to combine the information in these modalities using constraint analysis, for e.g., source localization from fMRI to inform EEG (Bonmassar et al., 2001, Liu et al., 1998, Bledowski et al., 2004), prediction of fMRI using the time dynamics of EEG as regressors (Eichele et al., 2005), and feature-based fusion that uses a common generative model to explain different modalities (Calhoun et al., 2006, Correa et al., 2008), where a feature is a lower dimensional representation of the brain function. Recently, effective techniques have been developed to acquire EEG data during fMRI data acquisition in the MR scanner (Goldman et al., 2000) and several approaches (Eichele et al., 2005, Eichele et al., 2008, Martinez-Montes et al., 2004, Moosmann et al., 2008, Debener et al., 2005) have been developed to analyze such concurrent recordings. These approaches study the trial-to-trial variability of single trial event-related potentials (ERPs) (Quian Quiroga and Garcia, 2003, Spencer, 2005) and its effects on hemodynamic activity (Stevens et al., 2005) to evaluate the relationship between the two modalities. In Eichele et al. (2005), single trial ERP data are shown to demonstrate paradigm induced amplitude modulation (AM) at certain time points and these AMs convolved with the hemodynamic response function (HRF) are used as predictors of fMRI responses. Similarly in Debener et al. (2005), the fMRI response was predicted based on the AMs observed in the temporal components obtained from independent component analysis (ICA) on the EEG data. As an extension of this approach, a spatio-temporal demixing of fMRI and EEG via parallel ICA on each dataset and subsequent data integration was presented in Eichele et al. (2008), wherein the time courses of the independent spatial components from fMRI were modeled with the HRF-convolved target stimulus as well as the detrended single trial weights of select independent temporal components from EEG. These approaches can be seen as extensions to the general linear model (GLM)-based approach for fMRI data analysis, where electrical activity from EEG is used to fit fMRI data, instead of or in addition to using the information about the hemodynamic activity related to the task. To summarize, data integration strategies for concurrently acquired data usually involve two stages: a form of feature extraction, e.g., source separation, followed by a cross-modality correlation or regression based on these features (Eichele et al., 2005, Eichele et al., 2008). Decomposing the data and then correlating across modalities may not ensure discovery of optimal relationship across modalities since this may lead to making non-physiological assumptions about the data and the cross-modality relationships (Martinez-Montes et al., 2004). Rather than performing source separation on each modality based on one criterion such as statistical independence of sources and then performing cross-modality integration based on a second criterion such as temporal correlation as done in Eichele et al. (2008), a more straightforward approach would be to unify the task by using a single cost for both purposes—evaluating the cross-modality associations and achieving source separation, both at the same time. Such a method would allow the study of the modular nature of brain function together with neuronal connectivity. Moreover, performing the fusion analysis at the group level is highly desirable since low cross-modality correlations as well as inter-subject differences may make it difficult to find one-to-one correspondence while matching the cross-modality linkages across subjects.

In this paper, we unify the two-step strategy in a multi-subject analysis by employing multi-set canonical correlation analysis (M-CCA) (Kettenring, 1971) to perform data fusion of concurrently acquired fMRI and EEG data from multiple subjects. Our fusion approach is based on a linear mixing model where each dataset is decomposed into a set of components (spatial areas for fMRI and temporal segments for EEG) and their mixing coefficients (weights reflecting the trial-to-trial variation of the components). We use M-CCA for finding a transformed coordinate system that maximizes the trial-to-trial covariation across the two modalities and across multiple subjects, and based on these covariation, we determine associations among the components across datasets. Thus, we perform source separation at the group level under one cost function that incorporates correlations across the modalities and the subjects. We identify regions in the fMRI data whose activation undergoes similar dynamics as the trial-to-trial amplitude modulations (AMs) in the EEG. Unlike univariate approaches for evaluating the association of each region with the AM at a certain latency, we can evaluate a subset of regions associated with the AM possibly manifested at a number of latencies. Consequently, our multivariate approach allows us to study localization of brain function as well as neuronal connectivity across the brain. The proposed fusion scheme is flexible and can be extended to jointly study different data types from a large number of subjects.

The superior performance of M-CCA in joint source separation for large number of fMRI datasets, its robustness to outliers, and its robustness to complex-valued data distributions, when compared with the competitive methods handling multiple datasets is shown in Li et al. (2009). A recently developed feature-based fusion approach uses M-CCA to link multiple data types through inter-subject covariation (Correa et al., 2008, Calhoun et al., 2006) to fuse three datasets: fMRI, EEG, and structural MRI obtained from subjects with schizophrenia and healthy controls (Correa et al., 2009). M-CCA identified a linearly transformed co-ordinate system such that the inter-subject covariations across different modalities are maximized. The results identified changes in the motor and temporal areas associated with the N2/P3 complex in the ERP, all of which have been well known to be affected in schizophrenia. However, in this previous application since the EEG-fMRI data were not acquired simultaneously, the method is limited to a feature-based analysis that cannot link the temporal dynamics of the fMRI dataset with the trial-to-trial variations of the EEG.

In order to test and compare our method, we use data consisting of a pattern learning paradigm of auditory tones presented at random and regular intervals. This paradigm stimulates subjective predictability/expectation which results in AMs in the EEG occurring at certain latencies, as shown in Eichele et al., 2005, Eichele et al., 2008). Our results demonstrate that M-CCA successfully identifies physiologically meaningful linked processes in both the fMRI and EEG data. M-CCA detects AM across the time on trial in the oddball task, which demonstrates long-term habituation in the multimodal data. The spatial areas linked with these AMs mainly include orbito-frontal cortex, motor areas, inferior-parietal gyrus, and temporal lobes. The EEG components show smooth oscillations; however, latency jitter and convolution with the HRF make it difficult to observe any prominent peaks.

Next we present a brief background on related fusion studies and applications of M-CCA in the next section, followed by the generative model for the M-CCA method in Theory section, our experiments in Materials and methods section, results in the fifth section, and discussion on the proposed method and results in the last section.

Section snippets

Background

Canonical correlation analysis (CCA) (Hotelling, 1936) and M-CCA—its extension to multiple datasets—are data-driven approaches that provide a natural framework for the study of two or more datasets. Recently, there has been increased interest in the use of CCA for fusion of features to obtain a more discriminating feature set in various pattern recognition applications (Sun et al., 2004, Yan et al., 2006, Xu and Mu, 2007, Shan et al., 2007, Sargın et al., 2007). Also, in biomedical

Theory

In this section, we present the proposed generative model for the fusion of concurrently acquired fMRI and EEG data from N subjects. Then, we explain the M-CCA procedure for fusion of the 2N datasets (two modalities for each subject).

Subjects

Fifteen healthy right-handed participants (21–28 years, seven female and eight male) took part in the experiment after providing a written statement of informed consent.

Stimuli

The stimuli during the sparse-sampling fMRI acquisition consisted of chords (50 ms) presented in eyes-closed condition via headphones (≈ 80 dB) with an onset synchrony of 2 s. Two tones were presented: 216 infrequent “targets” (500 Hz, 25% probability) were interspersed with frequent “standard” (250 Hz with 75% probability). The

Results

From the 25 component associations obtained through M-CCA, we focus on the two most correlated variations over trials as these showed maximum amplitude modulations as well as interesting spatial activation. For each cross-modality association, we discuss the trial-to-trial variations captured in the A vectors for both the modalities, the fMRI spatial component, and the EEG temporal component.

Discussion

We have introduced an effective data fusion approach to fuse single trial EEG data with fMRI data using M-CCA. The proposed approach offers a number of advantages over existing methods for analyzing concurrently acquired fMRI and EEG data. Being a data-driven approach, it minimizes the modeling assumptions about the cross-modality relationships between fMRI and EEG data. Its multivariate nature allows for inferences about connectivity across functional networks as opposed to univariate methods

Acknowledgment

This research was supported in part by the NIH grants R01 EB 005846 and R01 EB 000840, and the NSF grant 0612076.

References (41)

  • CalhounV.D. et al.

    Feature-based fusion of medical imaging data

    IEEE Trans. Inf. Technol. Biomed.

    (2009)
  • CalhounV.D. et al.

    A feature-based approach to combine functional MRI, structural MRI, and EEG brain imaging data

    Proc. EMBS

    (2006)
  • CorreaN.M. et al.

    Canonical correlation analysis for feature-based fusion of biomedical imaging modalities and its application to detection of associative networks in schizophrenia

    IEEE J. Sel. Top. Sign. Proc.

    (2008)
  • CorreaN.M. et al.

    Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis

  • DebenerS. et al.

    Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring

    J. Neurosci.

    (2005)
  • DelormeA. et al.

    EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics

    J. Neurosci. Methods

    (2004)
  • EicheleT. et al.

    Assessing the spatio-temporal evolution of neuronal activation with single-trial event-related potentials and functional MRI

    Proc. Nat. Acad. Sci. (PNAS)

    (2005)
  • FoxM.D. et al.

    The human brain is intrinsically organized into dynamic, anticorrelated functional networks

    Proc. Nat. Acad. Sci. (PNAS)

    (2005)
  • FoxM.D. et al.

    The global signal and observed anticorrelated resting state brain networks

    J. Neurophysiology

    (2009)
  • HallD.A. et al.

    “Sparse” temporal sampling in auditory fMRI

    Hum. Brain Mapp.

    (1999)
  • Cited by (121)

    View all citing articles on Scopus
    View full text