Reliability and statistical power analysis of cortical and subcortical FreeSurfer metrics in a large sample of healthy elderly

Highlights

• We assessed reliability and statistical power of surface-based morphometry.

• A large sample of healthy elderly was investigated.

• Several cortical and subcortical measures are reported.

• We provide a tool that allows researchers to perform their own power analysis.

• This will enable researchers to design well-powered studies.

Abstract

FreeSurfer is a tool to quantify cortical and subcortical brain anatomy automatically and noninvasively. Previous studies have reported reliability and statistical power analyses in relatively small samples or only selected one aspect of brain anatomy. Here, we investigated reliability and statistical power of cortical thickness, surface area, volume, and the volume of subcortical structures in a large sample (N = 189) of healthy elderly subjects (64 + years). Reliability (intraclass correlation coefficient) of cortical and subcortical parameters is generally high (cortical: ICCs > 0.87, subcortical: ICCs > 0.95). Surface-based smoothing increases reliability of cortical thickness maps, while it decreases reliability of cortical surface area and volume. Nevertheless, statistical power of all measures benefits from smoothing. When aiming to detect a 10% difference between groups, the number of subjects required to test effects with sufficient power over the entire cortex varies between cortical measures (cortical thickness: N = 39, surface area: N = 21, volume: N = 81; 10 mm smoothing, power = 0.8, α = 0.05). For subcortical regions this number is between 16 and 76 subjects, depending on the region. We also demonstrate the advantage of within-subject designs over between-subject designs. Furthermore, we publicly provide a tool that allows researchers to perform a priori power analysis and sensitivity analysis to help evaluate previously published studies and to design future studies with sufficient statistical power.

Introduction

There is a long history of manual morphometry investigating the structural changes occurring with age. This research is based on manually tracing brain regions of interest on magnetic resonance imaging (MRI) data. While this approach has led to tremendous insight regarding age-related changes in the brain, it also heavily relies on multiple intensively trained human raters. Standardizing all aspects of these procedures across labs often is difficult. This might give rise to undesired variations in results (Raz and Rodrigue, 2006). However, over the last decade, surface-based morphometry tools made it possible to non-invasively quantify gray matter in the human brain in a more automated fashion. Software packages such as FreeSurfer¹ or Caret² provide measurements of cortical and subcortical gray matter features based on MRI data. Because the algorithms that are implemented in these packages work with minimal user intervention, it became feasible to investigate large samples and apply this approach to a wide variety of research questions. Structural brain measurements (a) change systematically in development and aging (Fjell et al., 2013b, Hogstrom et al., 2013, Salat et al., 2004, Sowell et al., 2003), (b) reflect neuroplasticity in the context of experience and practice (Engvig et al., 2010), (c) can be important predictors of neurodegenerative disease (Dickerson et al., 2013, Gaser et al., 2013), and (d) have a genetic component (Joshi et al., 2011).

In the field of aging, subcortical volumes (Jäncke et al., 2014), as well as a variety of cortical measures are being analyzed, for instance, cortical thickness, surface area or volume (Fjell et al., 2014, Hogstrom et al., 2013, Mills and Tamnes, 2014). Analyzing a variety of parameters is warranted as they provide independent and complementary information about brain anatomy (Meyer et al., 2013, Winkler et al., 2010). However, age-related cortical thinning may render the reconstruction of cortical surfaces less reliable and reduce statistical power in the analysis of interest when studying older samples (e.g., to detect within-person individual change in longitudinal studies of aging). Pertinent studies (e.g., Han et al., 2006, Jovicich et al., 2006, Jovicich et al., 2013, Morey et al., 2010, Schnack et al., 2010, Wonderlick et al., 2009) cannot adequately contribute to this matter as the majority of them investigated test-retest reliability in young samples and relied on rather small sample sizes (often around 5 to 20 subjects). As demonstrated by Shoukri et al. (2004) and recently emphasized by Buchanan et al. (2014), a large sample size is necessary to compute reliability within an acceptable confidence interval. Further limitations of previous studies include the restriction to subcortical or regional cortical analyses, the restriction to only one measure (e.g., cortical thickness), or reporting reliability metrics that are difficult to interpret (for example, absolute difference of cortical thickness, which might correspond to different percent changes in two different regions).

In addition to the topic of reliability, the field of human neuroimaging recently increased its focus on the issue of statistical power (Button et al., 2013, Suckling et al., 2014). A major concern with neuroimaging studies is that relatively small sample sizes are not sufficient to detect relatively small differences between groups or conditions (Yarkoni et al., 2010). Therefore, performing an a priori power analysis to determine the appropriate sample size in the planning phase of a study is recommended. However, in the field of neuroimaging, calculation of statistical power is complicated by the fact that power is not uniform over the entire cortex, which results in different required sample sizes for different brain regions (Pardoe et al., 2013). While there exists a well-described statistical framework for performing statistical power analyses, only a minority of neuroimaging studies perform power analyses beforehand (Button et al., 2013). Recent work provided sample size calculations for cortical thickness studies (Pardoe et al., 2013). Building on this study, we extend this framework by including additional anatomical measures (cortical surface area, volume, and subcortical volume), other types of power analyses (post-hoc and sensitivity), and additional statistical tests (paired sample t-test).

The primary objective of the present study is therefore to investigate test-retest reliability and statistical power of cortical and subcortical measures derived with FreeSurfer. In detail, we aim to

• conduct our calculations using data from a large sample (N = 189);

• use a standardized and easy-to-interpret metric for reliability, namely, the intraclass correlation coefficient (ICC);

• assess several cortical measures (cortical thickness, surface area, volume) as well as measures of subcortical volume;

• provide vertex-wise information in order to assess regional variability in reliability and statistical power;

• investigate the effect of surface-based smoothing kernel size on reliability and statistical power;

• publicly provide surface data of reliability and statistical power in order for others to inquire about reliability and statistical power in brain regions they are most interested in;

• publicly provide a tool that allows others to perform various types of power analyzes (e.g., to determine the required sample size to detect an effect in cortical or subcortical measures before performing an experiment or to determine the sensitivity of a previously published study).

Note that the results presented here apply to the specific scanner type and acquisition sequence used in the study. Results might deviate for data from different scanner types and other acquisition schemes. Importantly, the power analysis tool we provide also allows researchers to base their power analysis on data previously acquired at their local scanner with their specific sequence. This, therefore, enables researchers to customize power analysis to their situation.