Research article
Valid statistical approaches for analyzing sholl data: Mixed effects versus simple linear models

https://doi.org/10.1016/j.jneumeth.2017.01.003Get rights and content

Highlights

  • In vivo studies of dendritic morphology in which multiple neurons are sampled per animal often use a simple linear model to detect significant differences which can lead to faulty inference.

  • Mixed models account for intra-class correlation that occurs with clustered data often generated in dendrite analysis to accurately estimate the standard deviation of the parameter estimate and, hence, produce accurate p-values.

  • A mixed effects approach accurately models the true variability in data sets sampling multiple neurons per animal, such as Sholl analysis.

Abstract

Background

The Sholl technique is widely used to quantify dendritic morphology. Data from such studies, which typically sample multiple neurons per animal, are often analyzed using simple linear models. However, simple linear models fail to account for intra-class correlation that occurs with clustered data, which can lead to faulty inferences.

New method

Mixed effects models account for intra-class correlation that occurs with clustered data; thus, these models more accurately estimate the standard deviation of the parameter estimate, which produces more accurate p-values. While mixed models are not new, their use in neuroscience has lagged behind their use in other disciplines.

Results

A review of the published literature illustrates common mistakes in analyses of Sholl data. Analysis of Sholl data collected from Golgi-stained pyramidal neurons in the hippocampus of male and female mice using both simple linear and mixed effects models demonstrates that the p-values and standard deviations obtained using the simple linear models are biased downwards and lead to erroneous rejection of the null hypothesis in some analyses.

Comparison with existing methods

The mixed effects approach more accurately models the true variability in the data set, which leads to correct inference.

Conclusions

Mixed effects models avoid faulty inference in Sholl analysis of data sampled from multiple neurons per animal by accounting for intra-class correlation. Given the widespread practice in neuroscience of obtaining multiple measurements per subject, there is a critical need to apply mixed effects models more widely.

Introduction

The central nervous system’s ability to process and distribute information relies on neural connectivity, and a key determinant of neural connectivity is the morphology of the dendrite (Libersat and Duch, 2004, Menon and Gupton, 2016, Scott and Luo, 2001). Altered dendritic morphology, including increased or decreased dendritic arborization are a shared feature of many neurodevelopmental disorders (NDDs) (Bourgeron, 2009, Fukuda et al., 2005, Garey, 2010, Keown et al., 2013, Supekar et al., 2013) and neurodegenerative diseases (Cochran et al., 2014, Kweon et al., 2016). Therefore the analysis of dendritic morphology is a critical tool in neuroscience studies.

A common problem encountered in neuroscience research is how to analyze complex dendritic structures. Sholl analysis is a method that has been widely used for decades to describe the complexity of neurons both from brain tissue sections and in vitro systems (Sholl, 1953), and it remains a key tool in neuroscience research for this application. In this method, concentric circles at specified radii (usually in 10 micrometer (μm) increments) are centered on the neuronal soma and the number of dendritic intersections at each circle is counted. Commonly reported endpoints from this analysis include the sum of all intersections within the Sholl radii, the number of intersections at individual radii, and the area under the curve for the whole or regions of the neuron (Ferreira et al., 2014, Gensel et al., 2010, Sholl, 1953). However, the statistical analysis of data generated using the Sholl technique is not consistent in the literature.

A common experimental design in studies of dendritic morphology, as well as many other studies in neuroscience, is that of multiple observations per subject, for example analyzing multiple neurons from one experimental animal. Simple linear models are commonly used to analyze these data; however, these models do not account for the clustered data structure of this experimental design, despite multiple reports over that last several years describing the issues associated with these approaches (Aarts et al., 2014, Boisgontier and Cheval, 2016, Ioannidis, 2005). While simple statistical methods such as t tests, Wilcoxon rank sum test, ANOVA, and regression are in wide-spread use, there are many situations in scientific research where the data structure violates the assumptions of these simple models. The effects of intra-subject correlation have been understood for several decades (Walsh, 1947). In the case where more than one observation is made on the same subject animal, for example, measurements on multiple neurons per animal, intra-subject correlation violates the assumption of complete independence of the observations. While in the last several years, the use of mixed effects models has increased in a variety of scientific and medical disciplines, neuroscience has lagged behind (Boisgontier and Cheval, 2016), despite publications warning of the sharp increase in faulty experimental designs, false positives, and spurious inferences that result (Aarts et al., 2014, Boisgontier and Cheval, 2016, Ioannidis, 2005).

As Aarts et al., 2014 point out, an increase in the number of neurons per animal gives the appearance of a large increase in power if a simple linear model is used. However, the true increase in power with increasing numbers of neurons per animal is relatively small, and this is only accounted for when the correct mixed effects model is used. Resources should be geared towards more animals rather than more neurons per animal. The reason for this is that data observed on neurons from the same animal are likely to have more in common with each other than with neurons from a different animal. Hence, an additional neuron from the same animal does not provide the same amount of additional information as another neuron from a different animal. When a simple linear model is fit to test for differences between treatments or other characteristics on two or more groups of animals, the variance is calculated under the assumption that each observation is independent of every other. The lower variability caused by similarity (dependence) between neurons from the same animal will result in an under-estimation of the within condition variance, which in turn results in an under-estimation of the p-value for the test of differences between conditions.

Many (Aarts et al., 2014, Galbraith et al., 2010, Senn, 1998) have shown through simulation and theoretical proofs that studies using simple linear models to analyze data with multiple measurements per subject have very high false positive rates. That is, if there is in fact no difference between the conditions, studies that do not adequately account for the clustered nature of the data will falsely yield a statistically significant result a large percentage of the time. Mixed effects models correctly model the clustering that results from measurements made on multiple neurons per animal and, hence, produce accurate p-values upon which inference is based. In the case of Sholl profile analysis where the number of intersections is measured at each radius, there is multi-level nesting (radii within neuron as well as neuron within animal) and an autoregressive covariance structure because measurements made at radii close to each other are likely to be more highly correlated than those far from each other. The commonly performed t test cannot accommodate either the multi-level nesting or the autoregressive covariance; in contrast, mixed effects models can accommodate both. One solution is a repeated measures analysis across radii, but this approach does not control for clustering due to multiple neurons per animal. Some authors (Wallin-Miller et al., 2016, Pawluski et al., 2012, Beauquis et al., 2013) have approached this problem by averaging the measurements across neurons to obtain one observation per animal at each radius, and then using a repeated measures analysis across radii. While this approach does not violate any model assumptions it does result in the loss of information about variability across neuron.

An additional complication in Sholl profile analysis is the multiple testing at multiple radii. Type one error inflation can be severe when multiple tests are performed. For example, when 10 radii are used, the probability of at least one type I error is about 40%. It is not uncommon for researchers performing Sholl profile analysis to fail to correct for global type I error inflation, and those who do often use methods that are either too severe, leading to an unnecessary loss of power, or too lenient, leading to a less than adequate control of error inflation. For example, using Bonferroni’s is too severe, while the Least Significant Difference correction for post-hoc comparisons is too lenient; more appropriate would be Scheffe’s (Neter et al., 1996, Zar, 1984). Here, we illustrate how to implement a method for controlling the false discovery rate in Sholl profile analysis, which addresses the issue of experiment-wise type I error inflation but is both powerful and accurate (Benjamini and Hochberg, 1995).

In this study, we review some of the issues involved in the analysis of clustered data, examine the misuse of the t test and the Wilcoxon rank sum test. We also compare the results of using mixed effects models versus simple linear models on real data from Sholl analyses of dendritic arborization of Golgi stained male and female wild type mouse hippocampal neurons to show how results of statistical analyses differ between the correct method (mixed effects models) and the incorrect method (simple linear models). Finally, we provide SAS® software code and annotated output for use of the mixed model in analyzing neuron architecture to simplify the analysis for non-statisticians.

Section snippets

Animals

All procedures and protocols were approved by the University of California, Davis Animal Care and Use Committee and were conducted in accordance with the NIH Guide for the Care and Use of Laboratory Animals. C57Bl/6J wild type mice were purchased from Jackson Labs (Bar Harbor, ME) and housed in clear plastic cages containing corn cob bedding. Mice were maintained on a 12 h light and dark cycle at 22 ± 2 °C. Feed (Diet 5058, LabDiet, St. Louis, MO) and water were available ad libitum.

Golgi staining and Sholl analysis

For this study

Results and discussion

Without the proper statistical method that accounts for the experimental design and data structure, the results of scientific research are questionable and non-reproducible. Without the ability to accurately quantify uncertainty and reproduce experimental results, scientists are not, in fact, meeting the demands of the scientific method for testing hypotheses. These issues have been recognized by the National Institutes of Health (Collins and Tabak, 2014, Landis et al., 2012, Pusztai et al.,

Conclusions

Here, we show an example data set that requires mixed effects model analysis and compare the results to common approaches that do not account for the clustered nature of the data. We show how both the standard error of the model parameters and the p-values are under-estimated, leading to faulty inference. We show an example of a common transformation that works well with these data to help normalize the distribution. We also show a method for analyzing Sholl profiles to account for both the

Disclosure statement

Authors have no conflicts of interest financial, personal, or other.

Acknowledgements

Research reported in this publication was supported by the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), through grant #UL1 TR000002, the National Institutes of Health (grants ES014901, ES011269, ES023513, U54 HD079125), and the United States Environmental Protection Agency (grant R833292) to PJL. The National Institute of Environmental Health Sciences (grant P30ES023513), and by the Eunice Kennedy Shriver National Institute Of Child Health &

References (34)

  • E. Aarts et al.

    A solution to dependency: using multilevel analysis to accommodate nested data

    Nat. Neurosci. Perspect.

    (2014)
  • H. Abdi
  • Y. Benjamini et al.

    Controlling the false discovery rate: a practical and powerful approach to multiple testing

    J. R. Stat. Soc. Ser. B (Methodol.)

    (1995)
  • H. Brown

    Applied Mixed Models in Medicine

    (2015)
  • F.S. Collins et al.

    Policy: NIH plans to enhance reproducibility

    Nature

    (2014)
  • S. Datta et al.

    Rank-sum tests for clustered data

    J. Am. Stat. Assoc.

    (2005)
  • T.A. Ferreira et al.

    Neuronal morphometry directly from bitmap images

    Nat. Methods

    (2014)
  • Cited by (0)

    View full text