- Research article
- Open Access
Independent external validation of cardiovascular disease mortality in women utilising Framingham and SCORE risk models: a mortality follow-up study
BMC Women's Health volume 14, Article number: 118 (2014)
We conducted an independent external validation of three cardiovascular risk score models (Framingham risk score model and SCORE risk charts developed for low-risk regions and high-risk regions in Europe) on a prospective cohort of 4487 Australian women with no previous history of heart disease, diabetes or stroke. External validation is an important step to evaluate the performance of risk score models using discrimination and calibration measures to ensure their applicability beyond the settings in which they were developed.
Ten year mortality follow-up of 4487 Australian adult women from the National Heart Foundation third Risk Factor Prevalence Study with no baseline history of heart disease, diabetes or stroke. The 10-year risk of cardiovascular mortality was calculated using the Framingham and SCORE models and the predictive accuracy of the three risk score models were assessed using both discrimination and calibration.
The discriminative ability of the Framingham and SCORE models were good (area under the curve > 0.85). Although all models overestimated the number of cardiovascular deaths by greater than 15%, the Hosmer-Lemeshow test indicated that the Framingham and SCORE-Low models were calibrated and hence suitable for predicting the 10-year cardiovascular mortality risk in this Australian population. An assessment of the treatment thresholds for each of the three models in identifying participants recommended for treatment were found to be inadequate, with low sensitivity and high specificity resulting from the high recommended thresholds. Lower treatment thresholds of 8.7% for the Framingham model, 0.8% for the SCORE-Low model and 1.3% for the SCORE-High model were identified for each model using the Youden index, at greater than 78% sensitivity and 80% specificity.
Framingham risk score model and SCORE risk chart for low-risk regions are recommended for use in the Australian women population for predicting the 10-year cardiovascular mortality risk. These models demonstrate good discrimination and calibration performance. Lower treatment thresholds are proposed for better identification of individuals for treatment.
Cardiovascular disease (CVD) is one of the leading causes of death worldwide and in Australia [1, 2]. There were 45622 CVD deaths recorded in 2011, more deaths occurred in Australian females (23755) than Australian males (21867) . Effective primary prevention strategies targeted at the individuals “at risk” are needed to reduce the number of CVD deaths . The recent American College of Cardiology and American Heart Association guidelines  stress that both primary and secondary prevention should be directed at those most likely to benefit.
Multivariable CVD risk score models enable the quantification of CVD risk. These models determine the probability of an individual experiencing a CVD event within a predefined time period by assessing the entire risk-factor profile . It identifies “at risk” individuals for intervention and is a cost-effective approach to CVD prevention . The 2010 American College of Cardiology Foundation and American Heart Association Guidelines  recommends all asymptomatic women to undergo a global CVD risk assessment. CVD risk factors for women include age, race/ethnicity, obesity, poor diet, excessive alcohol consumption, physical inactivity, smoking, high blood pressure, high cholesterol, diabetes, chronic kidney disease, genetics, coronary artery calcium and psychological factors, and also the interactions between some of these risk factors [8, 9]. It should be noted that not all of these risk variables have been incorporated into risk score models. Recommended treatment thresholds are used to identify individuals at increased CVD risk for treatment.
Commonly used risk score models and treatment thresholds include, the Framingham risk score model for 10-year CVD incidence or death  (20% treatment threshold) [11, 12], SCORE risk chart for 10-year CVD death (3%, 5%, 7% and 10% treatment thresholds) , Reynolds risk score model for 10-year CVD incidence and death , ASSIGN risk score model for 10-year CVD incidence and death (20% treatment threshold) , general CVD risk score model for 10-year CVD incidence and death  (10% and 20% thresholds) [16, 17], simplified general CVD risk score model for 10-year CVD incidence and death  (10% and 20% thresholds) [16, 17] and QRISK score model for 10-year CVD incidence and death . These risk score models were developed in the USA (Framingham, Reynolds and general CVD) and Europe (SCORE, ASSIGN and QRISK). Age, sex, systolic blood pressure (SBP), total cholesterol (TC) level, high-density lipoprotein cholesterol (HDL-C) level and smoking status were included in all six models. Diabetes status was included in all the models except the SCORE model. The Reynolds risk score model was the only model initially developed from a female population and it contains biomarkers in its calculation of the 10-year CVD risk . The QRISK score model includes more risk variables i.e. body mass index (BMI) (which is also found in a simpler version of the general CVD risk score model), family history (also found in the Reynolds and ASSIGN model), Townsend deprivation score (a measure of social deprivation is also found in the ASSIGN model), use of antihypertensive medication (also found in the general CVD model), self-assigned ethnicity, rheumatoid arthritis, chronic renal disease and atrial fibrillation, compared with other models .
The Framingham and SCORE models were selected and validated in this study as they have similar endpoints. Both models predict 10-year CVD death risk. In addition, the risk variables used to calculate the 10-year CVD risk for the Framingham and SCORE models were collected in the National Heart Foundation (NHF) Risk Factor Prevalence Study. Some risk score models did not have recommended treatment thresholds for identifying women at increased risk of CVD and their performance could not be assessed.
Performance of risk score models is typically overestimated in the original data which they were developed . External validation is an important step to evaluate the performance of risk score models using discrimination and calibration measures to ensure their applicability beyond the settings in which they were developed. The Framingham, SCORE and general CVD risk score models have been utilised to predict risk in an Australian population [20, 21]. Risk score models have also been developed for use in Australia i.e. from the Busselton Health Study and Dubbo Study. These models, however, are limited in applicability. The former model predicts the risk of coronary heart disease (CHD) hospitalisation or death while the latter predicts the CVD risk of older Australians aged 60 years and older [22, 23]. The National Vascular Disease Prevention Alliance recommends the use of the Framingham risk score model for estimating the 5-year CVD risk of disease-free Australians and an individual with a risk score of more than 15% is identified as high risk of developing CVD and is targeted for treatment .
The objective of this study was to conduct an independent external validation of the Framingham risk score model, SCORE model for low-risk regions (which was developed from European countries with low CVD rates: Belgium, Italy and Spain) and SCORE model for high-risk regions (European countries with high CVD rates: Denmark, Finland and Norway), using a mortality follow-up cohort of 4487 Australian women from the NHF Risk Factor Prevalence Study.
Participants were selected from the NHF third Risk Factor Prevalence Study, 1989 . Residents on the federal electoral rolls of December 1988 in North and South Sydney, Melbourne, Brisbane, Adelaide, Perth, Hobart, Darwin and Canberra were recruited for the Risk Factor Prevalence Study by systemic probability sampling of sex and 5-year age groups. A representative sample of 4487 women aged 20–69 years with no previous history of heart disease, diabetes or stroke at baseline were included for analysis. Participants taking medications to reduce their CVD risk factors were excluded.
Ethical approval for the NHF data was obtained in advance from the Australian Institute of Health Interim Ethics Committee, after consultation with the Commonwealth Privacy Commissioner. Participation was entirely voluntary. Those who participated signed an informed consent form . Participant information was anonymised prior to analysis. This study was approved by the Human Research Ethics Committee at Curtin University. The linkage and analysis of the NHF data with the National Death Index were approved by the current Ethics Committee of the Australian Institute of Health and Welfare (AIHW), and complies with the Declaration of Helsinki.
Risk factor measurements
A self-administered questionnaire was completed and information on demographic and clinical characteristics and smoking status were collected. Physical measurements were also taken: weight (to the nearest tenth of a kilogram), height (to the nearest centimetre), waist circumference and hip circumference were measured twice to the nearest centimetre according to standardised methodologies [26, 27], and systolic and diastolic blood pressure were recorded on the right arm of seated participants five minutes apart using mercury sphygmomanometers. The average of the two readings of blood pressure was used in the analysis . Fasting blood samples were also collected in EDTA tubes and despatched to the central laboratory at the Division of Clinical Chemistry, Institute of Medical and Veterinary Science, Adelaide each week for cholesterol levels to be assayed. Participants were classified as non-smokers, previous smokers or current smokers .
Cardiovascular disease outcomes
Mortality was ascertained to 31 December 1999 using the National Death Index maintained by the AIHW. The demographic data of the participants who were free from CVD and diabetes at baseline were submitted to AIHW. These data were then matched with the National Death Index using a probabilistic record linking package (“Automatch”). This provided data on the 10 year mortality follow-up. Causes of death were coded according to the International Classification of Diseases (ICD) 9th or 10th revision. ICD-9 codes 3900–4589 or ICD-10 codes I00.0-I99.9 were used for CVD deaths [28, 29]. The calculation of 10-year CVD risk from baseline data enabled assessment and comparison of the three risk score models and CVD death events were used because these events were determined, unlike non-fatal CVD events which are usually self-reported .
Risk score models
The Framingham risk score model predicts the 10-year risk of CVD death . It was developed using participants aged 30–74 years who were free of CVD and cancer from the American Framingham Heart Study. Age, sex, SBP, diastolic blood pressure, TC level, HDL-C level, smoking status and diabetes status were used in the calculation of the 10-year CVD risk . Although electrocardiogram-left ventricular hypertrophy (ECG-LVH) is included in the Framingham model for the calculation of the 10-year CVD risk, participants were assumed not to have LVH as ECG was not undertaken in the NHF third Risk Factor Prevalence Study. The most commonly used treatment threshold for the Framingham risk score model was 20% . This denotes that an individual who has a risk score of more than 20% is considered to be “at risk” of experiencing a CVD event within the next 10 years and should be targeted for treatment. Treatment includes lifestyle intervention (quit smoking, diet and exercise) as well as drug therapy (for lipids, hypertension and diabetes). Later versions of the Framingham risk score models include the National Cholesterol Education Program’s Third Adult Treatment Panel (ATP-III) risk assessment tool for the prediction of 10-year CHD risk, general CVD risk score model to predict the 10-year risk of CVD incidence and death and the Framingham risk prediction algorithm to quantify the 30-year risk of CVD [15, 30–33]. A new sex- and race-specific risk equation for predicting the 10-year atherosclerotic CVD risk, for fatal and non-fatal CVD events, for African-American and White men and women aged 40–79 years has also been published in the 2013 American College of Cardiology Foundation and American Heart Association Guidelines on the Assessment of Cardiovascular Risk .
The SCORE risk chart was developed by pooling 12 cohort studies in Europe. Participants aged 19–80 years with no previous history of heart attack were included in model development . Cohorts in Denmark, Finland and Norway were used to develop the SCORE risk chart for high-risk regions as they reported higher CVD rates, adjusted for risk factor levels, age , cohort sizes and data availability while cohorts in Belgium, Italy and Spain were used for low-risk regions . Fewer risk variables were used to calculate the 10-year risk of CVD death with the SCORE risk chart for low-risk and high-risk regions compared to the Framingham model. A larger cohort was used to develop the SCORE risk chart than the Framingham model. Age, sex, mean SBP, mean TC level, mean HDL-C level and smoking status were included in the calculation of the 10-year risk of CVD death for the SCORE models. Commonly used treatment thresholds for the SCORE risk chart included 3% and 5% for low-risk regions, and 3%, 5%, 7% and 10% for high-risk regions .
The data on the representative sample of 4487 Australian females were described using mean ± standard deviation or median (inter-quartile range) for continuous variables, while counts and percentages were used for categorical variables. Ten-year predicted CVD mortality risk was calculated for each participant using the Framingham and the two SCORE risk models and these were compared against observed actual CVD mortality. The predictive accuracy of the CVD risk score models were assessed using both discrimination and calibration.
Calibration was assessed by ranking the cohort into quintiles of risk and comparing the number of predicted and observed deaths within each quintile. The Hosmer-Lemeshow chi-squared goodness-of-fit test was used to measure the agreement between predicted and observed events across the quintiles of predicted risk for each of the models . A chi-squared p-value that is greater than 0.05 indicates a well-calibrated model. Graphically, calibration is represented as a plot of observed and predicted risk by quintiles of risk, arranged from low risk to high risk as determined by the CVD risk score models.
Discrimination was assessed by plotting the receiver operating characteristic (ROC) curve and calculating the area under the ROC curve (AUC) or c statistic. The ROC curve is a plot of true positives (sensitivity) against false positives (1-specificity) that provides a summary of sensitivity and specificity across a range of cut points for a continuous predictor . The c statistic refers to the probability that the predicted risk is higher for a case than for a non-case and a c statistic of 1 indicates perfect discrimination and the predicted risks are higher for all cases than non-cases even if the predicted risk score differs from the observed risk [36–38]. Sensitivity, specificity, likelihood ratio and Youden index  were calculated at the respective recommended treatment thresholds of the risk score models, and at the proposed cut-off. P-values of less than 0.05 were considered to be statistically significant. All statistical analyses were performed with IBM SPSS Statistics Version 21.
The demographic and clinical characteristics of the cohort of 4487 women without baseline heart disease, diabetes or stroke are presented in Table 1. The predicted 10-year CVD death risk calculated using the Framingham risk score model, and SCORE risk chart for low-risk and high-risk regions are also presented. There were 152 deaths from all causes and 28 deaths due to CVD observed during the 10 years of follow-up. These 152 deaths represented approximately 3.4% of the sample and 0.3% of the total deaths in the Australian female population in 1989 , at the start of the study, and 0.2% in 1999 , at the end of the 10-year follow-up.In Figure 1, the observed 10-year CVD death risk is compared with the Framingham, SCORE-Low and SCORE-High predicted risks across the range of ages in the cohort. The risk score models generally underestimated the 10-year CVD death risk in those below 50 years of age and overestimated the risk in those aged 50 years and above. The Framingham risk score model and SCORE risk chart for low-risk regions predicted similar risk levels in the age categories while the SCORE-High model predicted higher risk levels compared to the other two models.The Framingham, SCORE-Low and SCORE-High risk score models accurately predicted the 10-year risk of CVD death in the first four quintiles of risk (Figure 2). For the highest quintile of risk, however, all three models overestimated the 10-year risk. The overestimation was higher in the SCORE-High model than the Framingham and SCORE-Low models.
Table 2 presents the discrimination and calibration model performance statistics for the Framingham, SCORE-Low and SCORE-High models. The ratio of the 10-year CVD predicted risk to the observed risk in the highest quintile was 1.41 for the Framingham model, 1.35 for the SCORE-Low model and 2.02 for the SCORE-High model. The observed number of deaths is 28 (Table 2). The Framingham risk score model predicted the total number of deaths to be 34.5, an overestimation of approximately 23%. The SCORE-Low model predicted the total number of deaths to be 32.5, an overestimation of about 16% while the SCORE-High model overestimated the number of CVD deaths by approximately 75%. Calibration, as assessed using the Hosmer-Lemeshow test, indicated that the overestimation was not significant for the Framingham and SCORE-Low models and that the models were calibrated. All models reported high ROC AUC (≥0.858).
Cut-points of the risk score models were identified from a review of the literature and their respective sensitivity, specificity, likelihood ratio and Youden index are reported (Table 3). Low sensitivities were observed for the cut-points from the literature, with the Framingham risk score model reporting 0% sensitivity at the 20% cut-point. Lower cut-points were identified at the maximal Youden index. The proposed cut-points are 8.7% for the Framingham model, 0.8% for the SCORE-Low model and 1.3% for the SCORE-High model, to identify those at an increased 10-year CVD death risk, at greater than 78% sensitivity and 80% specificity.
In this cohort of 4487 Australian females, the Framingham, SCORE-Low and SCORE-High models performed well in discriminating those who died from CVD from those who did not. However, only the Framingham and SCORE-Low models indicated that they were well-calibrated (p > 0.05) in this population, although risk was generally overestimated. The calibration statistics reported in previous study also indicated that the Framingham model was well-calibrated, consistent with our study, but not the SCORE risk chart .
Overestimation was observed in the highest risk quintile in our study, where the ratio of the 10-year CVD predicted risk to the observed risk was 1.41 for the Framingham risk score model, 1.35 for the SCORE-Low model and 2.02 for the SCORE-High model. It had also been suggested previously that the accuracy of risk prediction at the extremes of risk needs to be improved . A systematic review study on the external validation of the Framingham risk score model for predicting the 10-year CHD or CVD risk reported that the ratio of 10-year CHD predicted to observed risk ranged from an under-prediction of 0.43 in a high-risk population to an over-prediction of 2.87 in a lower-risk population . There was a similar trend for the prediction of 10-year CVD risk . An under-prediction of risk was reported in Malaysian women using the SCORE risk chart .
In our study, the Framingham risk score model overestimated the total CVD deaths by 23%, the SCORE-Low model overestimated by about 16% and the SCORE-High model overestimated the number of CVD deaths by approximately 75%. In a study of Australian men free of heart disease, stroke and diabetes, the Framingham risk score model significantly over-predicted the risk of CHD and CVD death in all quintiles . A study on two German prospective cohorts found that the Framingham predicted risk substantially exceeded the observed risk in the cohorts, with an overestimation of at least 50% . An overestimation of risk was also observed in another study, the expected CHD deaths was 162, however, only 148 CHD deaths were observed . A study which compared the performance of the Framingham model and the SCORE model in predicting the 10-year CHD and CVD mortality risk in a non-diabetic population aged 40–65 years from a Spanish healthcare centre found that the SCORE model over-predicted the risk by 40% while the Framingham model over-predicted the risk by 64% (33% in men and 150% in women) . Using the SCORE risk chart for low-risk regions to predict the 10-year risk of CVD death, a study observed that the model over-estimated the CVD mortality risk in women . Overestimation was also reported for the SCORE risk chart for high-risk regions and the overestimation increased with age in women from Norway, consistent with our study findings . Similarly, the Framingham risk score model over-predicted the 5-year CVD risk (p < 0.05) in the Australian Diabetes, Obesity and Lifestyle study, evaluated against the general CVD risk score model . The risk was also overestimated using the general CVD risk score model. Another study assessed the performance of the Framingham risk score model and a general practice model in a cohort of 3582 women aged 60–79 years who were initially free of CHD in the British Women’s Heart and Health Study and observed that the Framingham model accurately predicted the 5-year CHD risk but over-predicted the CVD risk . Similar to our study, the model overestimated CVD risk in higher-risk groups. The Framingham risk score model also over-predicted the 10-year CVD risk in another study by 35% . It over-estimated the risk less than the ASSIGN model but more than the QRISK model. Its performance was also compared with the QRISK score model in two United Kingdom (UK) cohorts and the Framingham model over-predicted the risk as well .
Similar to our study findings, a systematic review study which assessed the performance of the Framingham risk score model using 25 validation cohorts of different population groups in the analysis reported that it accurately predicted CHD risk in populations from the United States, Australia and New Zealand . In the Dubbo study, the Framingham risk score model for CHD risk also accurately predicted the 10-year CHD incidence in men and women aged 60–79 years who were free of CVD or diabetes at baseline . Another study comparing the performance of ASSIGN, Framingham and general CVD risk score models in predicting the 10-year CVD risk in a UK cohort of patients from general practice reported that the Framingham model predicted the risk accurately but some over-prediction was observed in women of higher risk . The Framingham CVD model was also compared with the ATP-III score and Reynolds risk score using a case-cohort sample of the multi-ethnic Women’s Health Initiative Observational Cohort. Contrary to our study findings, the calibration performance of the Framingham CVD and ATP-III models were poor, however, the Reynolds model was calibrated .
It is important to extensively evaluate the applicability of risk models to each population, in order to avoid under-prediction or over-prediction . Possible causes of differences in estimation include differences in the populations used to develop the risk score models and different risk variables used to estimate risk. Generally, age, sex, SBP, TC level, HDL-C level and smoking status are included into risk score models. The incorporation of additional risk variables such as race/ethnicity, obesity, poor diet, excessive alcohol consumption, physical inactivity, chronic kidney disease, family history of CVD, coronary artery calcium and psychological factors, and also interactions between some of these risk factors could improve risk estimation in women. Recalibration has also been shown to improve the performance of risk score models after accounting for differences in the prevalence of risk factors and underlying CVD rates [58, 59]. Another option would be to quantify the amount of overestimation and adjust accordingly .
Existing studies generally reported good discrimination, consistent with our study findings. Our study reported AUC ≥ 0.858. High AUC (0.866) has also been reported for the Framingham risk score model in predicting CVD death in Australian men and women . AUC of 0.880 and 0.770 were also reported in women using the Framingham risk score model for predicting myocardial infarction and CHD death, in two German prospective studies . A study also found that the SCORE risk chart for low-risk regions for predicting the 10-year risk of CVD death discriminated Austrian women well (AUC = 0.780) . The discrimination performance of the SCORE risk chart for high-risk regions to predict the 10-year CVD mortality was also evaluated in Norway and the AUC values ranged from 0.660 to 0.720 in women . Another study which assessed the validity and utility of cardiovascular risk score models in a Malaysian population found that the Framingham risk score model and SCORE risk chart for low-risk and high-risk regions demonstrated good discrimination for predicting the 5-year cardiovascular mortality risk (AUC ranged from 0.758 to 0.763) in women .
The discriminative ability of the Framingham risk score model and SCORE risk chart have also been compared to other models. The Framingham risk score model was evaluated against the general CVD risk score model in the Australian Diabetes, Obesity and Lifestyle study and reported an AUC of 0.740, which is lower than the general CVD risk score model (AUC = 0.760) . Another study compared the performance of the Framingham, QRISK and ASSIGN models and reported that the Framingham model had lower ROC statistic (0.774) in women than QRISK and ASSIGN models . Similar results were observed in two UK cohorts with the Framingham risk score model reporting lower ROC statistics (approximately 0.776 and 0.760) than the QRISK score model . The performance of the Framingham CVD model was also compared with the ATP-III score and Reynolds risk score in a multi-ethnic Women’s Health Initiative Observational Cohort. The c statistics of the Framingham model and ATP-III score were 0.750 and 0.757, respectively, and were both lower than the Reynolds model .
The threshold discrimination is also an important aspect to consider, particularly in clinical practice . The sensitivity and specificity of CVD risk score models at different cut points or treatment thresholds should be reported . In our study, participants at increased risk of CVD were not adequately identified for treatment at the respective recommended treatment thresholds of the three risk score models. Low sensitivity and high specificity were reported as the recommended treatment thresholds were high. The higher treatment thresholds currently used for risk score models, under-treat individuals in higher risk groups. Our study identified lower treatment thresholds for the Framingham risk score model (8.7%) and SCORE risk chart for low- (0.8%) and high-risk (1.3%) regions, to improve the identification of individuals who require treatment with high sensitivity and specificity. Previous studies have also reported that it is difficult for individuals to exceed the recommended treatment thresholds even with markedly elevated risk factors. The Framingham risk score model was compared against the ASSIGN risk score using the Scottish Heart Health Extended Cohort. At the 20% cut point, the sensitivity of the Framingham risk score model was low (45.6%), while the specificity was 82.5% . In another study, the Framingham model for predicting the 10-year CHD risk classified only 33% of all CHD events occurred in women as high risk, using a threshold of 20% . The Framingham risk score model and SCORE model to predict CVD mortality was also evaluated in the Netherlands and the Framingham model only identified 0.7% of the population for treatment while the SCORE model assigned 0.4% for treatment, at a threshold of 10% . Among those identified for treatment using the Framingham and SCORE models, their observed risk was 6.2% and 10.2%, respectively, indicating that the recommended threshold was inadequate for identifying individuals for treatment. The 10-year predicted risk was < 10% or low across age groups and risk factor levels using the ATP-III risk assessment tool . Similarly, all men < 30 years with substantial risk factor burden using the Framingham risk score model also reported low risk using existing threshold . In addition, the 10-year predicted risk was also found to be < 10% in most men < 50 years and most women < 70 years . An approach to lower treatment thresholds would seem appropriate. A new threshold of ≥ 10% has been recommended for identifying women at high CVD risk for treatment as it is difficult for a woman below 75 years of age with several markedly elevated risk factors to exceed a 10% (let alone a 20%) with the ATP-III risk assessment tool . The most recent guidelines released by the American College of Cardiology and American Heart Association Task Force on Practice Guidelines have recommended an even lower threshold of 7.5% for the 10-year risk of atherosclerotic CVD for identifying individuals for treatment . Lower treatment thresholds, as proposed, would improve the identification of individuals who require treatment, at the expense of increasing the numbers to treat and increasing associated costs. Determining thresholds using diagnostic measures to maximise sensitivity and specificity is still the preferred approach.
Study strengths and limitations
Our study has limitations. The endpoints of our study were fatal CVD events established from death certificates and no incidence data was available. There were only 28 fatal CVD events among 4487 women in the 10-year follow-up. It is possible that calibration defaults are present due to differences in the populations used to develop the Framingham and SCORE models. Though differences exist, these models have continued to exhibit calibration when externally validated against other populations and have been used extensively in the assessment of CVD risk in previous studies [20, 22, 54, 55]. Participants were assumed not to have LVH as ECG was not undertaken in the NHF third Risk Factor Prevalence Study and this could affect risk prediction of the Framingham risk score model. It has been reported that while baseline ECG data to determine LVH were not available, this is unlikely to have a significant impact on risk prediction as ECG-diagnosed LVH is very rare in the general population without CVD . Only one set of baseline measurements was recorded for some risk variables but important variables were measured twice. Menopausal status of women was not recorded at time of survey. Low number of CVD events limits the ability for further detailed analyses to assess other risk factors currently not included in the risk score models discussed.
In this study, we have provided an independent external validation of the Framingham risk score model and the SCORE risk chart for low-risk and high-risk regions on a cohort of 4487 Australian females. The use of the Framingham risk score model and SCORE risk chart for low-risk regions, for predicting the 10-year CVD mortality risk in the Australian women population is recommended. These models demonstrate good discrimination and calibration performance. Lower treatment thresholds are needed, in order to better identify individuals for treatment.
Coronary heart disease
National Heart Foundation
Australian Institute of Health and Welfare
International Classification of Diseases
Systolic blood pressure
Electrocardiogram-left ventricular hypertrophy
National Cholesterol Education Program’s Third Adult Treatment Panel
Receiver operating characteristic curve
Area under the curve
Body mass index
Waist to hip ratio
High-density lipoprotein cholesterol
Framingham 10-year predicted risk for CVD death
SCORE-Low 10-year predicted risk for CVD death
SCORE-High 10-year predicted risk for CVD death
- Proposed* :
Cut-point identified from our study using the Youden index for high sensitivity and specificity.
World Health Organization: Global Status Report on Noncommunicable Diseases 2010. Book Global Status Report on Noncommunicable Diseases. 2011, Geneva: WHO
3303.0 Causes of Death, Australia, 2011. http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/3303.02011?OpenDocument.
Michos ED, Blumenthal RS: How accurate are 3 risk prediction models in US women?. Circulation. 2012, 125: 1723-1726. 10.1161/CIRCULATIONAHA.112.099929.
Goff DC, Lloyd-Jones DM, Bennett G, O’Donnell CJ, Coady S, Robinson J, D’Agostino RBS, Schwartz JS, Gibbons R, Shero ST, Greenland P, Smith SC, Lackland DT, Sorlie P, Levy D, Stone NJ, Wilson PWF: ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2013, 2014: 63.
Australian Institute of Health and Welfare: Book Cardiovascular disease: Australian facts 2011. Cardiovascular Disease Series. Cat. no. CVD 53. 2011, Canberra: AIHW, Cardiovascular Disease: Australian Facts 2011. Cardiovascular Disease Series. Cat. no. CVD 53, Cardiovascular Disease Series.
Barzi F, Patel A, Gu D, Sritara P, Lam TH, Rodgers A, Woodward M, Asia Pacific Cohort Studies Collaboration: Cardiovascular risk prediction tools for populations in Asia. J Epidemiol Community Health. 2007, 61: 115-121.
Greenland P, Alpert JS, Beller GA, Benjamin EJ, Budoff MJ, Fayad ZA, Foster E, Hlatky MA, Hodgson JM, Kushner FG, Lauer MS, Shaw LJ, Smith SC, Taylor AJ, Weintraub WS, Wenger NK: ACCF/AHA Guideline for Assessment of Cardiovascular Risk in Asymptomatic Adults: A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines Developed in Collaboration With the American Society of Echocardiography, American Society of Nuclear Cardiology, Society of Atherosclerosis Imaging and Prevention, Society for Cardiovascular Angiography and Interventions, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance. Circulation. 2010, 2010 (122): e584-e636.
Australian Institute of Health and Welfare: Book Women and Heart Disease: Cardiovascular Profile of Women in Australia. Cardiovascular Disease Series no. 33. Cat. no. CVD 49. 2010, Canberra: AIHW, Women and Heart Disease: Cardiovascular Profile of Women in Australia. Cardiovascular Disease Series no. 33. Cat. no. CVD 49, Cardiovascular Disease Series no. 33. Cat. no. CVD 49.
Goh LGH, Dhaliwal SS, Lee AH, Bertolatti D, Della PR: Utility of established cardiovascular disease risk score models for the 10-year prediction of disease outcomes in women. Expert Rev Cardiovasc Ther. 2013, 11: 425-435. 10.1586/erc.13.26.
Anderson KM, Odell PM, Wilson PW, Kannel WB: Cardiovascular disease risk profiles. Am Heart J. 1991, 121: 293-298. 10.1016/0002-8703(91)90861-B.
Neil HAW, Perera R, Armitage JM, Farmer AJ, Mant D, Durrington PN: Estimated 10-year cardiovascular risk in a British population: results of a national screening project. Int J Clin Pract. 2008, 62: 1322-1331. 10.1111/j.1742-1241.2008.01828.x.
Woodward M, Brindle P, Tunstall-Pedoe H, for the SIGN group on risk estimation: Adding social deprivation and family history to cardiovascular risk assessment: the ASSIGN score from the Scottish Heart Health Extended Cohort (SHHEC). Heart. 2007, 93: 172-176.
Conroy RM, Pyörälä K, Fitzgerald AP, Sans S, Menotti A, De Backer G, De Bacquer D, Ducimetière P, Jousilahti P, Keil U, Njølstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM: Estimation of ten-year risk of fatal cardiovascular disease in Europe: The SCORE project. Eur Heart J. 2003, 24: 987-1003. 10.1016/S0195-668X(03)00114-3.
Ridker PM, Buring JE, Rifai N, Cook NR: Development and validation of improved algorithms for the assessment of global cardiovascular risk in women. JAMA. 2007, 297: 611-619. 10.1001/jama.297.6.611.
D’Agostino RBS, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB: General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008, 117: 743-753. 10.1161/CIRCULATIONAHA.107.699579.
Mosca L, Benjamin EJ, Berra K, Bezanson JL, Dolor RJ, Lloyd-Jones DM, Newby LK, Piña IL, Roger VL, Shaw LJ, Zhao D, Beckie TM, Bushnell C, D’Armiento J, Kris-Etherton PM, Fang J, Ganiats TG, Gomes AS, Gracia CR, Haan CK, Jackson EA, Judelson DR, Kelepouris E, Lavie CJ, Moore A, Nussmeier NA, Ofili E, Oparil S, Ouyang P, Pinn VW, et al: Effectiveness-based guidelines for the prevention of cardiovascular disease in women—2011 update: a guideline from the American Heart Association. J Am Coll Cardiol. 2011, 57: 1404-1423. 10.1016/j.jacc.2011.02.005.
Genest J, McPherson R, Frohlich J, Anderson T, Campbell N, Carpentier A, Couture P, Dufour R, Fodor G, Francis GA, Grover S, Gupta M, Hegele RA, Lau DC, Leiter L, Lewis GF, Lonn E, Mancini GBJ, Ng D, Pearson GJ, Sniderman A, Stone JA, Ur E: Canadian Cardiovascular Society/Canadian guidelines for the diagnosis and treatment of dyslipidemia and prevention of cardiovascular disease in the adult – 2009 recommendations. Can J Cardiol. 2009, 2009 (25): 567-579.
Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, Minhas R, Sheikh A, Brindle P: Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ. 2008, 336: 1475-1482. 10.1136/bmj.39609.449676.25.
Altman DG, Royston P: What do we mean by validating a prognostic model?. Stat Med. 2000, 19: 453-473. 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5.
Chen L, Tonkin AM, Moon L, Mitchell P, Dobson A, Giles G, Hobbs M, Phillips PJ, Shaw JE, Simmons D, Simons LA, Fitzgerald AP, De Backer G, De Bacquer D: Recalibration and validation of the SCORE risk chart in the Australian population: the AusSCORE chart. Eur J Cardiovasc Prev Rehabil. 2009, 16: 562-570. 10.1097/HJR.0b013e32832cd9cb.
Zomer E, Owen A, Magliano DJ, Liew D, Reid C: Validation of two Framingham cardiovascular risk prediction algorithms in an Australian population: the ‘old’ versus the ‘new’ Framingham equation. Eur J Cardiovasc Prev Rehabil. 2011, 18: 115-120.
Simons LA, Simons J, Palaniappan L, Friedlander Y, McCallum J: Risk functions for prediction of cardiovascular disease in elderly Australians: the Dubbo Study. Med J Aust. 2003, 178: 113-116.
Knuiman MW, Vu HTV, Bartholomew HC: Multivariate risk estimation for coronary heart disease: the Busselton Health Study. Aust N Z J Public Health. 1998, 22: 747-753. 10.1111/j.1467-842X.1998.tb01487.x.
National Vascular Disease Prevention Alliance: Guidelines for the Assessment of Absolute Cardiovascular Disease Risk. Book Guidelines for the Assessment of Absolute Cardiovascular Disease Risk. 2009, City
Australian Risk Factor Prevalence Study Management Committee: Survey No. 3 1989. Book Survey No. 3 1989. 1990, Canberra: National Heart Foundation of Australia and Australia Institute of Health
Boyle CA, Dobson AJ, Egger G, Benault SA: Waist-to-hip ratios in Australia: a different picture of obesity. Aust J Nutr Diet. 1993, 50: 57-64.
Alexander H, Dugdale A: Which waist-hip ratio?. Med J Aust. 1990, 153: 367-368.
World Health Organization: International Classification of Diseases. Manual of the International Statistical Classification of Diseases, Injuries and Causes of Death, 9th Revision (ICD-9). Book International Classification of Diseases. Manual of the International Statistical Classification of Diseases, Injuries, and Causes of Death, 9th Revision (ICD-9). 1977, Geneva: WHO
World Health Organization: International Classification of Diseases. Manual of the International Statistical Classification of Diseases, Injuries and Causes of Death, 10th Revision (ICD-10). Book International Classification of Diseases. Manual of the International Statistical Classification of Diseases, Injuries and Causes of Death, 10th Revision (ICD-10). 1998, Geneva: WHO
Expert Panel on Detection Evaluation and Treatment of High Blood Cholesterol in Adults: Executive summary of the third report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III). JAMA. 2001, 285: 2486-2497. 10.1001/jama.285.19.2486.
National Cholesterol Education Program (NCEP) Expert Panel on Detection Evaluation and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III): Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) Final Report. NIH Publication No. 02–5215. Circulation. 2002, 106: 3143-3421.
Cavanaugh-Hussey MW, Berry JD, Lloyd-Jones DM: Who exceeds ATP-III risk thresholds? Systematic examination of the effect of varying age and risk factor levels in the ATP-III risk assessment tool. Prev Med. 2008, 47: 619-623. 10.1016/j.ypmed.2008.07.012.
Pencina MJ, D’Agostino RBS, Larson MG, Massaro JM, Vasan RS: Predicting the 30-year risk of cardiovascular disease: the Framingham heart study. Circulation. 2009, 119: 3078-3084. 10.1161/CIRCULATIONAHA.108.816694.
Sans S, Kesteloot H, Kromhout D, on behalf of the Task Force: The burden of cardiovascular diseases mortality in Europe: Task Force of the European Society of Cardiology on Cardiovascular Mortality and Morbidity Statistics in Europe. Eur Heart J. 1997, 18: 1231-1248. 10.1093/oxfordjournals.eurheartj.a015434.
Hosmer DW, Lemeshow S: Applied Logistic Regression. 2000, John Wiley & Sons, Inc: NJ, USA
Cook NR: Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007, 115: 928-935. 10.1161/CIRCULATIONAHA.106.672402.
Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982, 143: 29-36. 10.1148/radiology.143.1.7063747.
Cui J: Overview of risk prediction models in cardiovascular disease research. Ann Epidemiol. 2009, 19: 711-717. 10.1016/j.annepidem.2009.05.005.
Youden WJ: Index for rating diagnostic tests. Cancer. 1950, 3: 32-35. 10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3.
Commonwealth of Australia: 3302.0 - Deaths, Australia, 1993. Book 3302.0 - Deaths, Australia, 1993. 1994, Canberra: Australian Bureau of Statistics
Australian Bureau of Statistics: 3302.0 - Deaths, 1999. Book 3302.0 - Deaths, 1999. 2000, Canbera: Australian Bureau of Statistics
Orford JL, Sesso HD, Stedman M, Gagnon D, Vokonas P, Gaziano JM: A comparison of the Framingham and European Society of Cardiology coronary heart disease risk prediction models in the normative aging study. Am Heart J. 2002, 144: 95-100. 10.1067/mhj.2002.123317.
Brindle P, Beswick A, Fahey T, Ebrahim S: Accuracy and impact of risk assessment in the primary prevention of cardiovascular disease: a systematic review. Heart. 2006, 92: 1752-1759. 10.1136/hrt.2006.087932.
Selvarajah S, Kaur G, Haniff J, Cheong KC, Hiong TG, van der Graaf Y, Bots ML: Comparison of the Framingham Risk Score, SCORE and WHO/ISH cardiovascular risk prediction models in an Asian population. Int J Cardiol. 2014, 176: 211-218. 10.1016/j.ijcard.2014.07.066.
Dhaliwal SS, Welborn TA: Central obesity and multivariable cardiovascular risk as assessed by the Framingham prediction scores. Am J Cardiol. 2009, 103: 1403-1407. 10.1016/j.amjcard.2008.12.048.
Hense H-W, Schulte H, Löwel H, Assmann G, Keil U: Framingham risk function overestimates risk of coronary heart disease in men and women from Germany — results from the MONICA Augsburg and the PROCAM cohorts. Eur Heart J. 2003, 24: 937-945. 10.1016/S0195-668X(03)00081-2.
Thomsen TF, McGee D, Davidsen M, Jørgensen T: A cross-validation of risk-scores for coronary heart disease mortality based on data from the Glostrup Population Studies and Framingham Heart Study. Int J Epidemiol. 2002, 31: 817-822. 10.1093/ije/31.4.817.
Barroso LC, Muro EC, Herrera ND, Ochoa GF, Hueros JIC, Buitrago F: Performance of the Framingham and SCORE cardiovascular risk prediction functions in a non-diabetic population of a Spanish health care centre: a validation study. Scand J Prim Health Care. 2010, 28: 242-248. 10.3109/02813432.2010.518407.
Ulmer H, Kollerits B, Kelleher C, Diem G, Concin H: Predictive accuracy of the SCORE risk function for cardiovascular disease in clinical practice: a prospective evaluation of 44 649 Austrian men and women. Eur J Cardiovasc Prev Rehabil. 2005, 12: 433-441. 10.1097/01.hjr.0000174791.47059.80.
Lindman AS, Veierod MB, Pedersen JI, Tverdal A, Njolstad I, Selmer R: The ability of the SCORE high-risk model to predict 10-year cardiovascular disease mortality in Norway. Eur J Cardiovasc Prev Rehabil. 2007, 14: 501-507. 10.1097/HJR.0b013e328011490a.
May M, Lawlor DA, Brindle P, Patel R, Ebrahim S: Cardiovascular disease risk assessment in older women: can we improve on Framingham? British Women’s Heart and Health prospective cohort study. Heart. 2006, 92: 1396-1401. 10.1136/hrt.2005.085381.
Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P: Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: Prospective open cohort study. BMJ. 2007, 335: 136-10.1136/bmj.39261.471806.55.
Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, Brindle P: Performance of the QRISK cardiovascular risk prediction algorithm in an independent UK sample of patients from general practice: a validation study. Heart. 2008, 94: 34-39. 10.1136/hrt.2007.134890.
Eichler K, Puhan MA, Steurer J, Bachmann LM: Prediction of first coronary events with the Framingham score: a systematic review. Am Heart J. 2007, 153: 722-731. 10.1016/j.ahj.2007.02.027. e728
de la Iglesia B, Potter JF, Poulter NR, Robins MM, Skinner J: Performance of the ASSIGN cardiovascular disease risk score on a UK cohort of patients from general practice. Heart. 2011, 97: 491-499. 10.1136/hrt.2010.203364.
Cook NR, Paynter NP, Eaton CB, Manson JE, Martin LW, Robinson JG, Rossouw JE, Wassertheil-Smoller S, Ridker PM: Comparison of the Framingham and Reynolds risk scores for global cardiovascular risk prediction in the multiethnic Women’s health initiative / clinical perspective. Circulation. 2012, 125: 1748-1756. 10.1161/CIRCULATIONAHA.111.075929.
Hense H-W: Observations, predictions and decisions—assessing cardiovascular risk assessment. Int J Epidemiol. 2004, 33: 235-239. 10.1093/ije/dyh118.
D’Agostino RBS, Grundy S, Sullivan LM, Wilson P, for the CHD Risk Prediction Group: Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001, 286: 180-187. 10.1001/jama.286.2.180.
Liu J, Hong Y, D’Agostino RBS, Wu Z, Wang W, Sun J, Wilson PWF, Kannel WB, Zhao D: Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese multi-provincial cohort study. JAMA. 2004, 291: 2591-2599. 10.1001/jama.291.21.2591.
Dhaliwal SS, Welborn TA: Central obesity and cigarette smoking are key determinants of cardiovascular disease deaths in Australia: a public health perspective. Prev Med. 2009, 49: 153-157. 10.1016/j.ypmed.2009.07.019.
Cooney MT, Dudina A, D’Agostino R, Graham IM: Cardiovascular risk-estimation systems in primary prevention. Circulation. 2010, 122: 300-310. 10.1161/CIRCULATIONAHA.109.852756.
Jackson R: Cardiovascular risk prediction: are we there yet?. Heart. 2008, 94: 1-3. 10.1136/hrt.2007.138040.
Koller MT, Steyerberg EW, Wolbers M, Stijnen T, Bucher HC, Hunink MGM, Witteman JCM: Validity of the Framingham point scores in the elderly: results from the Rotterdam study. Am Heart J. 2007, 154: 87-93. 10.1016/j.ahj.2007.03.022.
Scheltens T, Verschuren WMM, Boshuizen HC, Hoes AW, Zuithoff NP, Bots ML, Grobbee DE: Estimation of cardiovascular risk: a comparison between the Framingham and the SCORE model in people under 60 years of age. Eur J Cardiovasc Prev Rehabil. 2008, 15: 562-566. 10.1097/HJR.0b013e3283063a65.
Berry JD, Lloyd-Jones DM, Garside DB, Greenland P: Framingham risk score and prediction of coronary heart disease death in young men. Am Heart J. 2007, 154: 80-86. 10.1016/j.ahj.2007.03.042.
Ford ES, Giles WH, Mokdad AH: The distribution of 10-Year risk for coronary heart disease among U.S. adults: Findings from the National Health and Nutrition Examination Survey III. J Am Coll Cardiol. 2004, 43: 1791-1796. 10.1016/j.jacc.2003.11.061.
Milne R, Gamble G, Whitlock G, Jackson R: Framingham Heart Study risk equation predicts first cardiovascular event rates in New Zealanders at the population level. N Z Med J. 2003, 116: U662.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6874/14/118/prepub
The authors declare that they have no competing interests.
LGHG was involved in drafting the manuscript, performing the analysis, interpreting the data and revising the manuscript critically for important intellectual content. TAW participated in the study design, acquired the data and revised the manuscript critically for important intellectual content. SSD conceived the study, performed the analysis and data interpretation and revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript.
About this article
Cite this article
Goh, L.G.H., Welborn, T.A. & Dhaliwal, S.S. Independent external validation of cardiovascular disease mortality in women utilising Framingham and SCORE risk models: a mortality follow-up study. BMC Women's Health 14, 118 (2014). https://doi.org/10.1186/1472-6874-14-118
- Cardiovascular disease
- Risk estimation
- Model performance
- Primary prevention