Abstract
Deriving statistical models to predict one variable from one or more other variables, or predictive modeling, is an important activity in obesity and nutrition research. To determine the quality of the model, it is necessary to quantify and report the predictive validity of the derived models. Conducting validation of the predictive measures provides essential information to the research community about the model. Unfortunately, many articles fail to account for the nearly inevitable reduction in predictive ability that occurs when a model derived on one data set is applied to a new data set. Under some circumstances, the predictive validity can be reduced to nearly zero. In this overview, we explain why reductions in predictive validity occur, define the metrics commonly used to estimate the predictive validity of a model (for example, coefficient of determination (R2), mean squared error, sensitivity, specificity, receiver operating characteristic and concordance index) and describe methods to estimate the predictive validity (for example, cross-validation, bootstrap, and adjusted and shrunken R2). We emphasize that methods for estimating the expected reduction in predictive ability of a model in new samples are available and this expected reduction should always be reported when new predictive models are introduced.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Heshka S, Feld K, Yang MU, Allison DB, Heymsfield SB . Resting energy expenditure in the obese: a cross-validation and comparison of prediction equations. J Am Diet Assoc 1993; 93: 1031–1036.
Tzoulaki I, Liberopoulos G, Ioannidis JP . Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 2009; 302: 2345–2352.
Baan CA, Ruige JB, Stolk RP, Witteman JC, Dekker JM, Heine RJ et al. Performance of a predictive model to identify undiagnosed diabetes in a health care setting. Diabetes Care 1999; 22: 213–219.
Dixon JB, Chuang LM, Chong K, Chen SC, Lambert GW, Straznicky NE et al. Predicting the glycemic response to gastric bypass surgery in patients with type 2 diabetes. Diabetes Care 2013; 36: 20–26.
Forns X, Ampurdanes S, Llovet JM, Aponte J, Quinto L, Martinez-Bauer E et al. Identification of chronic hepatitis C patients without hepatic fibrosis by a simple predictive model. Hepatology 2002; 36: 986–992.
Hayes MT, Hunt LA, Foo J, Tychinskaya Y, Stubbs RS . A model for predicting the resolution of type 2 diabetes in severely obese subjects following Roux-en Y gastric bypass surgery. Obes Surg 2011; 21: 910–916.
Li S, Zhao JH, Luan J, Luben RN, Rodwell SA, Khaw KT et al. Cumulative effects and predictive value of common obesity-susceptibility variants identified by genome-wide association studies. Am J Clin Nutr 2010; 91: 184–190.
Thomas DM, Ivanescu AE, Martin CK, Heymsfield SB, Marshall K, Bodrato VE et al. Predicting successful long-term weight loss from short-term weight-loss outcomes: new insights from a dynamic energy balance model (the POUNDS Lost study). Am J Clin Nutr 2015; 101: 449–454.
Cancello R, Tordjman J, Poitou C, Guilhem G, Bouillot JL, Hugol D et al. Increased infiltration of macrophages in omental adipose tissue is associated with marked hepatic lesions in morbid human obesity. Diabetes 2006; 55: 1554–1561.
Chen H, Sullivan G, Quon MJ . Assessing the predictive accuracy of QUICKI as a surrogate index for insulin sensitivity using a calibration model. Diabetes 2005; 54: 1914–1925.
Clasey JL, Bradley KD, Bradley JW, Long DE, Griffith JR . A new BIA equation estimating the body composition of young children. Obesity 2011; 19: 1813–1817.
Garcia AL, Wagner K, Hothorn T, Koebnick C, Zunft HJ, Trippo U . Improved prediction of body fat by measuring skinfold thickness, circumferences, and bone breadths. Obes Res 2005; 13: 626–634.
Huang TT, Watkins MP, Goran MI . Predicting total body fat from anthropometry in Latino children. Obes Res 2003; 11: 1192–1199.
Rush EC, Chandu V, Plank LD . Prediction of fat-free mass by bioimpedance analysis in migrant Asian Indian men and women: a cross validation study. Int J Obes (Lond) 2006; 30: 1125–1131.
Russell M, Mendes N, Miller KK, Rosen CJ, Lee H, Klibanski A et al. Visceral fat is a negative predictor of bone density measures in obese adolescent girls. J Clin Endorinol Metab 2010; 95: 1247–1255.
Kuhn M, Johnson K . Applied Predictive Modeling. Springer, 2013.
Copas JB . Regression, prediction and shrinkage. J R Stat Soc Ser B Methodol 1983; 45: 311–354.
Schmid M, Riganti-Fulginei F, Bernabucci I, Laudani A, Bibbo D, Muscillo R et al. SVM versus MAP on accelerometer data to distinguish among locomotor activities executed at different speeds. Comput Math Methods Med 2013; 343084: 1–7.
Hitchcock C, Sober E . Prediction versus accommodation and the risk of overfitting. Brit J Philos Sci 2004; 55: 1–34.
Harrell FE Jr, Lee KL, Mark DB . Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361–387.
Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; 98: 691–698.
Collins GS, Reitsma JB, Altman DG, Moons KG . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med 2015; 13: 1–10.
Heymsfield SB, Thomas D, Bosy-Westphal A, Shen W, Peterson CM, Muller MJ . Evolving concepts on adjusting human resting energy expenditure measurements for body size. Obes Rev 2012; 13: 1001–1014.
Ley SH, Hamdy O, Mohan V, Hu FB . Prevention and management of type 2 diabetes: dietary components and nutritional strategies. Lancet 2014; 383: 1999–2007.
Mittlbock M, Schemper M . Explained variation for logistic regression. Stat Med 1996; 15: 1987–1997.
Zou KH, O'Malley AJ, Mauri L . Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 2007; 115: 654–657.
Ozenne B, Subtil F, Maucort-Boulch D . The precision-recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J Clin Epidemiol 2015; 68: 855–859.
Harrell FFJ . Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer: New York, 2001.
Hastie T, Tibshirani R, Friedman J . The Elements of Statistical Learning. Springer, 2009.
James G, Witten D, Hastie T, Tibshirani R . An Introduction to Statistical Learning. Springer, 2013.
Efron B, Tibshirani RJ . An Introduction to the Bootstrap vol. 57. Chapman & Hall/CRC Monographs on Statistics & Applied Probability: New York, 1993.
Ambroise C, McLachlan GJ . Selection bias in gene extraction on the basis of microarray gene-expression data. Proc Natl Acad Sci USA 2002; 99: 6562–6566.
Molinaro AM, Simon R, Pfeiffer RM . Prediction error estimation: a comparison of resampling methods. Bioinformatics 2005; 21: 3301–3307.
Bobko P . Correlation and Regression: Applications for Industrial Organizational Psychology and Management. Sage Publications, 2001.
Cattin P . Estimation of the predictive power of a regression model. J Appl Psychol 1980; 65: 407–414.
Yamanaka N, Okamoto E, Kuwata K, Tanaka N . A multiple regression equation for prediction of posthepatectomy liver failure. Ann Surg 1984; 200: 658–663.
Scalfi L, Marra M, De Filippo E, Caso G, Pasanisi F, Contaldo F . The prediction of basal metabolic rate in female patients with anorexia nervosa. Int J Obes Relat Metab Disord 2001; 25: 359–364.
Mahon AD, Marjerrison AD, Lee JD, Woodruff ME, Hanna LE . Evaluating the prediction of maximal heart rate in children and adolescents. Res Q Exerc Sport 2010; 81: 466–471.
Roediger HL 3rd, Watson JM, McDermott KB, Gallo DA . Factors that determine false recall: a multiple regression analysis. Psychon Bull Rev 2001; 8: 385–407.
Nomani MZ, Khan AH, Shahda MM, Nomani AK, Sattar SA . Predicting serum gastrin levels among men during Ramadan fasting. East Mediterr Health J 2005; 11: 119–125.
McKeon JL, Murree-Allen K, Saunders NA . Prediction of oxygenation during sleep in patients with chronic obstructive lung disease. Thorax 1988; 43: 312–317.
Puyau MR, Adolph AL, Vohra FA, Zakeri I, Butte NF . Prediction of activity energy expenditure using accelerometers in children. Med Sci Sports Exerc 2004; 36: 1625–1631.
Blalock SJ, Currey SS, DeVellis RF, Anderson JJ, Gold DT, Dooley MA . Using a short food frequency questionnaire to estimate dietary calcium consumption: a tool for patient education. Arthritis Care Res 1998; 11: 479–484.
Siervo M, Prado C, Hooper L, Munro A, Collerton J, Davies K et al. Serum osmolarity and haematocrit do not modify the association between the impedance index (Ht(2)/Z) and total body water in the very old: the Newcastle 85+ study. Arch Gerontol Geriatr 2015; 60: 227–232.
Hoffman DJ, Toro-Ramos T, Sawaya AL, Roberts SB, Rondo P . Estimating total body fat using a skinfold prediction equation in Brazilian children. Ann Hum Biol 2012; 39: 156–160.
Lee JJ, Freeland-Graves JH, Pepper MR, Yao M, Xu B . Predictive equations for central obesity via anthropometrics, stereovision imaging and MRI in adults. Obesity 2014; 22: 852–862.
Jackson AS, Stanforth PR, Gagnon J, Rankinen T, Leon AS, Rao DC et al. The effect of sex, age and race on estimating percentage body fat from body mass index: The Heritage Family Study. Int J Obes Relat Metab Disord 2002; 26: 789–796.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138.
Apfelbacher CJ, Loerbroks A, Cairns J, Behrendt H, Ring J, Kramer U . Predictors of overweight and obesity in five to seven-year-old children in Germany: results from cross-sectional studies. BMC Public Health 2008; 8: 171.
Gerhard GS, Benotti P, Wood GC, Chu X, Argyropoulos G, Petrick A et al. Identification of novel clinical factors associated with hepatic fat accumulation in extreme obesity. J Obes 2014; 368210: 1–8.
Xi B, Zhao X, Shen Y, Wu L, Hou D, Cheng H et al. An obesity genetic risk score predicts risk of insulin resistance among Chinese children. Endocrine 2014; 47: 825–832.
Little RJA, Rubin DB . Statistical Analysis with Missing Data. Wiley: New York, 1987.
Graham JW . Missing data analysis: making it work in the real world. Annu Rev Psychol 2009; 60: 549–576.
Tibshirani R . Regression shrinkage and selection via the Lasso. J R Stat Soc Ser B Methodol 1996; 58: 267–288.
de los Campos G, Gianola D, Allison DB . Predicting genetic predisposition in humans: the promise of whole-genome markers. Nat Rev Genet 2010; 11: 880–886.
Cotillard A, Poitou C, Duchateau-Nguyen G, Aron-Wisnewsky J, Bouillot J-L, Schindler T et al. Type 2 diabetes remission after gastric bypass: what is the best prediction tool for clinicians? Obes Surg 2015; 25: 1128–1132.
Acknowledgements
This study was supported in part by NIH grants R25DK099080, R25HL124208 and P30DK056336. We gratefully acknowledge the anonymous reviewers for their helpful suggestions, which substantially improved this article.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no conflict of interest.
Additional information
Disclaimer
The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.
Rights and permissions
About this article
Cite this article
Ivanescu, A., Li, P., George, B. et al. The importance of prediction model validation and assessment in obesity and nutrition research. Int J Obes 40, 887–894 (2016). https://doi.org/10.1038/ijo.2015.214
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/ijo.2015.214
This article is cited by
-
Validation of multiparametric MRI based prediction model in identification of pseudoprogression in glioblastomas
Journal of Translational Medicine (2023)
-
Covariate dependent Markov chains constructed with gradient boost modeling can effectively generate long-term predictions of obesity trends
BMC Research Notes (2023)
-
Machine learning for nuclear cardiology: The way forward
Journal of Nuclear Cardiology (2019)
-
Prediction of uncomplicated pregnancies in obese women: a prospective multicentre study
BMC Medicine (2017)
-
Accuracy of BMI correction using multiple reports in children
BMC Obesity (2016)