Skip to main content

Evaluating sex as a biological variable in preclinical research: the devil in the details

Abstract

Translating policy into action is a complex task, with much debate surrounding the process whereby US and Canadian health funding agencies intend to integrate sex and gender science as an integral component of methodological rigor and reporting in health research. Effective January 25, 2016, the US National Institutes of Health implemented a policy that expects scientists to account for the possible role of sex as a biological variable (SABV) in vertebrate animal and human studies. Applicants for NIH-funded research and career development awards will be asked to explain how they plan to factor consideration of SABV into their research design, analysis, and reporting; strong justification will be required for proposing single-sex studies. The Canadian Institutes of Health Research is revising their peer review accreditation process to ensure that peer reviewers are skilled in applying a critical lens to protocols that should be incorporating sex and gender science. The current paper outlines the components that peer reviewers in North America will be asked to assess when considering whether SABV is appropriately integrated into research designs, analyses, and reporting. Consensus argues against narrowly defining rules of engagement in applying SABV, with criteria provided for reviewers as guidance only. Scores will not be given for each criterion; applications will be judged on the overall merit of scientific innovation, rigor, reproducibility, and potential impact.

Background

In May 2014, the US National Institutes of Health announced a policy aimed at integrating sex as a biological variable (SABV) into biomedical research [1]. The announcement led to a spirited debate in the scientific literature and popular press on the subject of promoting due diligence in exploring sex differences while respecting the principles of scientific freedom [25]. Our objective here is to describe the decisions taken by two major North American health funding agencies, the US National Institutes of Health (NIH) and the Canadian Institutes of Health Research (CIHR), for applying metrics to evaluate SABV by peer reviewers in their respective organizations [1, 6].

Main text

In order to determine the preferred scoring method to gauge the appropriateness of SABV in research protocols involving preclinical studies with vertebrate animal and humans, an iterative process was undertaken by both the NIH and CIHR. The NIH’s Office of Research on Women’s Health championed the process in the USA, with SABV being considered as only one element of a broad NIH initiative to enhance rigor, transparency, and reproducibility of preclinical biomedical research [7]. The Institute of Gender and Health championed SABV on behalf of the CIHR, as part of the Canadian federal government’s Health Portfolio policy on sex-and-gender-based analysis [8]. Two independent champions, two different policies, yet the selection of metrics for the evaluation of SABV ended up being remarkably similar for both funding agencies. The process consisted of (a) an environmental scan of the literature for critical components of SABV; (b) key informant consultations and stakeholder forums; and (c) high-level discussions by decision-makers inside each funding organization. In February 2015, the US Office of Research on Women’s Health and Canada’s Institute of Gender and Health compared notes. They then created a consensus list of 13 evaluation criteria that might be used as a minimal standard for peer reviewers evaluating SABV information.

As part of the feedback process on the evaluation criteria, the Organization for the Study of Sex Differences (OSSD) anonymously surveyed health researchers who had registered for its 2015 annual meeting on their perceptions of the importance of including each of these elements in a peer review rating scale for SABV. Figure 1 illustrates the proportion of online survey respondents (n = 30) who assessed each of the 13 evaluation criteria as being absolutely critical, rather critical, important but not critical, or somewhat important/not critical in the assessment of SABV. Seventy-five researchers then participated in a stakeholder’s forum held at the OSSD meeting on April 21, 2015, in Palo Alto, CA, to discuss and vote on issues that generated the most diverse responses in the initial survey. Participants were international, originating from Canada, South America, Europe, and Australia, with a majority representation from the USA. Almost three quarters of respondents considered themselves well versed in the study of sex differences. Sixty-seven percent stated they studied sex differences for 6 years or more. The distribution of responses shown in Fig. 1 is found to be representative of the response of the larger group of 75 people at the stakeholder’s forum.

Fig. 1
figure 1

Ranking of the 13 SABV evaluation criteria according to the respondents’ assessment of absolutely critical or rather critical

The majority of participants proposed that peer reviewers should use the evaluation criteria as a SABV checklist guide only and that individual scores not be assigned to each criterion. Respondents expressed a strong preference for incorporating the SABV rating into the overall scientific evaluation of the research protocol without asking peer reviewers to provide a separate SABV score. Only one third of participants preferred that the quality of SABV be rated independently but still be factored into the overall evaluation score.

These results were considered in parallel to other stakeholder consultations and independent discussion with decision-makers at the NIH and CIHR. The verdict fell concurrently for both organizations that no specific score would be attributed to the integration of SABV in research protocols. Consensus argued against narrowly defining rules of engagement in applying SABV, as disagreement exists even within the OSSD community on gold standard criteria for assessment. SABV evaluation criteria will be promoted as a guide for peer reviewers when assessing the overall scientific merit of the application, as regards scientific innovation, rigor, reproducibility, and expected impact. The CIHR’s reviewer checklist can be found online [9]. Table 1 illustrates the key questions peer reviewers are encouraged to ask when determining the overall score for the application.

Table 1 Questions reviewers should consider when evaluating SABV

To assist peer reviewers in acquiring competency to evaluate SABV, the CIHR’s Institute of Gender and Health developed an online training module in September 2015 to enable critical appraisal of the appropriate integration, or omission, of SABV in research protocols [10]. This module also aims to fill a gap in knowledge and skills for the 60–80 % of the CIHR biomedical researchers who, during mandatory reporting, indicated that they do not integrate sex or gender into their research designs [11]. The NIH’s Office of Research on Women’s Health similarly held a method workshop and posted an online report to assist scientists debuting the integration of SABV in their research programs [12].

Concerns remain that integrating SABV into research programs may be costly, that resources may not be available, and that duplication of all experimental procedures in both sexes may be onerous and burdensome. If the NIH and CIHR were truly advocating that equal numbers of males and females be included in all studies, then this concern would be valid. However, the NIH and CIHR advocate no such blind change in protocol. Instead, researchers and peer reviewers are being asked to thoughtfully consider whether a single-sex study is justified when research results are to be applied to both sexes. Funding organizations recognize that new, potentially transformative discoveries such as one by a Canadian researcher showing that chronic pain is mediated by different immune cells in males and females will only be brought about by funding studies that attempt to explain sexually dimorphic discrepancies in the epidemiology of disease or response to treatment [13]. By using scientific acumen, researchers are given the freedom to decide if and how SABV may be relevant to their field of study. Will the answer sometimes require researchers to ask for larger budgets for their studies to include equal numbers of males and females? Absolutely. However, if well justified, both the NIH and CIHR believe that the investment in balancing the sexes will be worthwhile.

The topic of statistical power necessary to detect sex differences continues to generate controversy, as does the criticism that policies linked to SABV ignore the contribution of other social determinants of health such as gender, education, and socioeconomic status as risk factors for disease [1416]. Both the NIH and CIHR recognize that animal models are not the panacea to cure all human suffering. Both organizations simultaneously fund clinical, health systems and population health research to address these questions. With respect to sample size, what is clear is that sample size and power for preclinical studies ultimately depend on the specific question being posed by the researcher. It would be inherently challenging to uniformly implement sample size requirements for SABV across the research spectrum. Discretion will therefore be left to the researcher to justify the sample size required for any given research approach using SABV. Peer reviewers will be asked to weigh in on whether the calculations make sense, in the same way that sample size has been evaluated in the past. No specific score will be recorded for this element.

Conclusion

The NIH and CIHR intend that peer reviewers should be given the instruction to make sure SABV is accounted for in research design, analysis, and reporting and, if only one sex will be used, that this is well justified. Evaluation of SABV is expected to depend on the best judgment of the reviewers along with other criteria used to assess the scientific excellence of a given proposal. The research question at hand and the specific scientific context should guide appropriate consideration of SABV in a project, with the understanding that transparency is crucial to building a sex-specific evidence base.

Abbreviations

CIHR:

Canadian Institutes of Health Research

NIH:

National Institutes of Health

OSSD:

Organization for the Study of Sex Differences

SABV:

sex as a biological variable

References

  1. Clayton JA, Collins FS. Policy: NIH to balance sex in cell and animal studies. Nature. 2014;509(7500):282–3.

    Article  PubMed  Google Scholar 

  2. McCullough LD, de Vries GJ, Miller VM, Becker JB, Sandberg K, McCarthy MM. NIH initiative to balance sex of animals in preclinical studies: generative questions to guide policy, implementation, and metrics. Biol Sex Differ. 2014;5:15. doi:10.1186/s13293-014-0015-5.

    Article  PubMed Central  PubMed  Google Scholar 

  3. Sandberg K, Umans JG. Georgetown Consensus Conference Work Group. Recommendations concerning the new U.S. National Institutes of Health initiative to balance the sex of cells and animals in preclinical research. FASEB J. 2015;29(5):1646–52. doi:10.1096/fj.14-269548.

    Article  CAS  PubMed  Google Scholar 

  4. McCarthy MM. Incorporating sex as a variable in preclinical neuropsychiatric research. Schiozpr Bull. 2015;41(5):1016–20.

    Article  Google Scholar 

  5. Klein SL, Schiebinger L, Stefanick ML, Cahill L, Danska J, de Vries GJ, et al. Opinion: sex inclusion in basic research drives discovery. Proc Natl Acad Sci U S A. 2015;112(17):5257–8.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  6. Johnson JL, Beaudet A. Sex and gender reporting in health research: why Canada should be a leader. Can J Public Health. 2012;104(1):e80–1.

    PubMed  Google Scholar 

  7. NOT-OD-15-103: enhancing reproducibility through rigor and transparency. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-103.html. Accessed 13 January 2016.

  8. Health Canada Health Portfolio Sex and Gender-based Analysis Policy. 2009. Available at http://www.hc-sc.gc.ca/hl-vs/pubs/women-femmes/sgba-policy-politique-ags-eng.php Accessed Dec 2, 2015.

  9. Criteria that will be used to for evaluating the integration of sex and gender into CIHR strategic funding initiatives—Biomedical & Translational Research. Available at http://www.cihr-irsc.gc.ca/e/49337.html. Printable checklist available at http://www.cihr-irsc.gc.ca/e/documents/igh-checklist-integrating-fund-initiatives-bio-en.pdf . Accessed 2 Dec 2015

  10. Canadian Institutes of Health Research, Institute of Gender and Health. Online Training Modules: Integrating Sex & Gender in Health Research. Available at http://www.cihr-irsc.gc.ca/igh-competency.html. Accessed Dec 2, 2015.

  11. Johnson J, Sharman Z, Vissandjée B, Stewart DE. Does a change in health research funding policy related to the integration of sex and gender have an impact? PLoS One. 2014;9(6):e99900. doi:10.1371/journal.pone.0099900.eCollection2014.

    Article  PubMed Central  PubMed  Google Scholar 

  12. Methods and techniques for integrating the biological variable sex into preclinical research. http://orwh.od.nih.gov/sexinscience/researchtrainingresources/methodstechniquesbiovar.asp. Accessed 13 January 2016.

  13. Sorge RE, Mapplebeck JC, Rosen S, et al. Different immune cells mediate mechanical pain hypersensitivity in male and female mice. Nat Neurosci. 2015;18(8):1081–3.

    Article  CAS  PubMed  Google Scholar 

  14. Richardson SS, Reiches M, Shattuck-Heidorn H, Labonte ML, Consoli T. Opinion: focus on preclinical sex differences will not address women’s and men’s health disparities. Proc Natl Acad Sci U S A. 2015;112(44):13419–20.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  15. Johnson JL, Greaves L, Repta R. Better science with sex and gender: facilitating the use of a sex and gender-based analysis in health research. Int J Equity Health. 2009;8:14. doi:10.1186/1475-9276-8-14. Published online 2009 May 6.

    Article  PubMed Central  PubMed  Google Scholar 

  16. Braveman P, Gottlieb L. The social determinants of health: It’s time to consider the causes of the causes. Public Health Rep. 2014;129 Suppl 2:19–31.

    PubMed Central  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Cara Tannenbaum.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

All authors contributed equally to the design and conduct of the study. CT and JMS performed the analysis. All authors contributed equally to the interpretation of the findings and the drafting of the manuscript. All authors read and approved the final version for submission.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tannenbaum, C., Schwarz, J.M., Clayton, J.A. et al. Evaluating sex as a biological variable in preclinical research: the devil in the details. Biol Sex Differ 7, 13 (2016). https://doi.org/10.1186/s13293-016-0066-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13293-016-0066-x

Keywords