Abstract
Objective. To assess the validity of self-reports by patients with systemic sclerosis (SSc) of 5 common, chronic conditions (hypertension, diabetes, cancer, depression, and osteoarthritis/back pain) as compared to chart review.
Methods. SSc patients at a large referral hospital self-reported on a number of comorbidities. Their inpatient and outpatient medical records were abstracted using a standardized data extraction form. Sensitivity, specificity, and positive predictive value of the self-reported diagnoses were calculated using data from the chart review as gold standard.
Results. Self-reported comorbidity data were verified by chart review for 130 patients with SSc. The sensitivity of the self-reports for the 5 comorbid conditions was low [range 35% (cancer) to 86% (diabetes)]. The specificity was moderate to high [range 76% (osteoarthritis/back pain) to 99% (cancer)]. The positive predictive values ranged from 31% (depression) to 86% (cancer).
Conclusion. Self-reports of comorbidities do not provide optimal data for the identification of common, chronic conditions in patients with SSc.
Systemic sclerosis (SSc) is a severe, chronic, multisystem disorder characterized by fibrosis of skin and internal organs1. It is associated with significant morbidity, including disfiguring skin thickening, finger ulcers, joint contractures, pulmonary hypertension, interstitial lung disease, esophageal dysmotility, gastrointestinal vascular ectasias and bleeding, chronic diarrhea, and renal failure. SSc is associated with poor outcomes including functional disability2 and impaired health-related quality of life3,4. Overall survival is considerably decreased. In a recent study by Mayes, et al5, absolute survival after diagnosis ranged from 77.9% at 5 years to 26.8% at 20 years (median survival about 11 yrs), and relative survival (ratio of observed to expected survival) was as low as 35% at 20 years, compared to age, sex, and race matched individuals. There are very few studies attempting to estimate costs associated with SSc. One study reported the annual direct and indirect costs of SSc in the United States to be ~$1.5 billion6.
Comorbidity has been defined as “any distinct additional entity that has existed or may occur during the clinical course of a patient who has the index disease under study”7. Thus, when dealing with patients with a particular illness, it refers to diseases other than the index disease. Comorbidity is a powerful predictor of health outcomes costs8–13. It can also be a confounder or an effect modifier in the association between disease and outcome14. Outcomes of importance in longitudinal studies of systemic sclerosis (SSc) include function, health-related quality of life, costs, and mortality. Although some studies have documented a high rate of other autoimmune diseases15 and chronic disease burden16 among patients with SSc, no studies have specifically assessed the effect of comorbidities on the outcomes of interest in these patients. Moreover, it is essential to measure comorbidities because these may not improve with therapy and should be taken into account in any interventional study. Clearly, the presence and severity of comorbidities require systematic study to evaluate their full role in the outcome of SSc.
Some studies have shown that patients can accurately self-report their current and past medical conditions, including comorbidities17–19. Recently, the Self-Administered Comorbidity Questionnaire (SCQ)20 was developed. The SCQ lists common medical conditions, including heart disease, lung disease, kidney disease, hypertension, diabetes, cancer, depression, osteoarthritis (OA), and back pain. The patient indicates for each condition if it is present, being treated, and/or imposes functional limitation. The SCQ was found to be reliable and valid and to correlate with health status and service utilization in patients admitted to general medical and surgical wards.
Given the importance of measuring comorbidities in SSc and of doing so efficiently, we asked patients in the Canadian Scleroderma Research Group (CSRG) registry to self-report comorbidities using the SCQ. However, in a previous study of the construct validity of the SCQ in our SSc patients, we found that SSc patients were not able to distinguish comorbidities from their index disease, in particular heart, lung, and kidney disease21. Thus, including an SCQ score in a statistical model to control for comorbidities could result in over-adjusting for disease severity. Thus, we concluded that the SCQ as a whole was not a useful measure of comorbidity in patients with SSc.
Nevertheless, we identified 5 common conditions included in the SCQ: hypertension, diabetes, cancer, depression, and OA/back pain, that we believe to be important comorbidities in SSc, i.e., conditions generally distinct from the index condition (namely, SSc), but that could have considerable effects on outcomes of interest in SSc and that patients would likely be able to distinguish from their manifestations of SSc. We investigated the validity of self-reports of these conditions in SSc, by comparing them to chart review.
MATERIALS AND METHODS
Study sample
The CSRG Registry is a well established database funded by the Canadian Institutes of Health Research. Currently, over 800 patients with SSc from 15 centers across Canada are enrolled. Of these, over 100 are followed for their regular medical care at the SMBD-Jewish General Hospital in Montreal. For this study, we reviewed the inpatient and outpatient files of the CSRG patients followed at this single institution.
All patients in the CSRG Registry are recruited from the practices of rheumatologists across Canada. They must have a diagnosis of SSc made by the referring rheumatologist, be > 18 years of age, be fluent in English or French, and be likely to comply with study procedures and visits. The patients included in this study were those whose baseline visit was between September 2004 and August 2007.
Patients in the CSRG Registry undergo an extensive standardized evaluation, including obtaining data on demographic variables, a yearly history and physical examination by a physician, and laboratory testing. They also complete a series of self-administered instruments, including the SCQ.
Study instrument
The original SCQ20 lists common medical conditions, including high blood pressure, diabetes, cancer, depression, OA, and back pain; patients are asked to indicate for each condition if it is present. For our patients, the original instructions for the SCQ were modified to emphasize that the goal of the instrument was to identify “other” problems aside from the patient’s SSc. Permission to use and translate the SCQ was obtained from the original authors. The SCQ was translated into Canadian French based on published methodology22 and administered to all patients in the language of their choice.
Procedure for chart extraction
A trained chart extractor, blinded to the patients’ SCQ reports, performed detailed chart review of the patients’ in and outpatient files at the SMBD-Jewish General Hospital. Using a standardized data extraction form, the chart extractor identified any physician diagnoses of high blood pressure, diabetes (defined as a diagnosis of diabetes or diabetic complication such as diabetic retinopathy, nephropathy, or foot ulcers), cancer (including skin cancers, solid tumors, metastases, lymphoma, or leukemia), depression, and OA (of any joints) or back pain. The relevant sources of data to document the above included medical notes of the treating rheumatologist or medical notes from other physicians involved in the patient’s care and available in the patient’s in and outpatient files. Charts were reviewed without time limit.
Statistical analysis
Results of self-reported conditions were compared to results of the chart review, using the chart review as the gold standard. Sensitivity, specificity, and positive predictive value were calculated for each disease condition separately using 2 × 2 tables. Sensitivity was defined as the proportion of persons who reported having a specific disease condition on the SCQ confirmed by chart review. Specificity was calculated as the proportion of persons who did not report having a specific disease condition on the SCQ and for whom chart review did not identify that condition. Positive predictive value was defined as the proportion of persons who had the disease confirmed by chart review among those who reported having the disease.
Ethical considerations
The CSRG data collection protocol was approved by the Ethics Committee of McGill University. Patients included in the Registry provided written, informed consent. This study was approved by the ethics review board of the SMBD-Jewish General Hospital, Montreal, Canada.
RESULTS
Our study included 130 patients, of which 92% were female, mean age was 58 (± 14) years, 35% had diffuse SSc, mean disease duration since the onset of the first non-Raynaud’s disease manifestation was 9 (± 8) years, and 45% had some education beyond high school.
The sensitivity of the self-reports for the 5 named comorbid conditions ranged from 35% (cancer) to 86% (diabetes) (Table 1). All of the conditions, except diabetes, had sensitivity less than or equal to 65%. The specificity ranged from 76% (OA/back pain) to 99% (cancer). All of the conditions except OA/back pain and hypertension had specificity above 90%. The positive predictive values ranged from 31% (depression) to 86% (cancer).
DISCUSSION
In our study of patients with SSc, we found strikingly low sensitivities, fair to high specificities, and a range of positive predictive values for self-reports of 5 common chronic conditions. The sensitivity and positive predictive values are the most important characteristics of self-reports, the former because it reflects the true positive rate of comorbidity in the population (e.g., chart review identified 17 patients with cancer, of which only 6 self-reported this) and the latter because it reflects how reliable the self-reports are (e.g., among the 13 patients who reported having depression, chart review confirmed this in only 4). Specificity, or the true negative rate, is of less importance because it is the presence, rather than the absence of disease that is of particular interest in the assessment of comorbidity. Thus, the low sensitivities for all conditions except diabetes and low positive predictive values for hypertension, depression, and OA/back pain found in this study suggest that patient self-reports may not be the best method to identify comorbidities in SSc patients.
Several hypotheses could explain the poor results. For high blood pressure and cancer, the low sensitivity could result from the fact that patients are being treated for these conditions and thus may currently have normal blood pressure, or they may be cured or in remission from their cancers, prompting an answer of “no” to whether they have the condition. Diagnoses of depression, confirmed by chart review, were grossly under-reported by patients (sensitivity 40%) whereas many reports of depression were not confirmed by chart review (positive predictive value 31%). These inconsistencies could be explained by patients’ reluctance to admit to having a mental health problem and/or by differences in interpretation of the term depression, with patients referring to the symptom and physicians to the diagnosis. OA and back pain were the most common diagnoses reported by both patients and physicians and yet had low sensitivity and positive predictive value. Again, this could result from patients reporting symptoms and physicians diagnoses.
The validity of self-reported diagnoses of various chronic conditions has been the subject of numerous investigations, and results have varied considerably depending on the condition of interest and the population under study. For example, using data from the Women’s Health Initiative, Margolis, et al found that self-reports of “treated” diabetes were concordant with medication inventories in almost 80% of participants and concluded that these were sufficiently accurate to allow their use in epidemiologic studies23. However, the population in that study was registered nurses, a generally better educated and health conscious group. Nevertheless, agreement between self-reports and chart review has generally been found to be good for well known diagnoses such as diabetes, hypertension, and cancer23–26, but poor for conditions with less clear diagnostic criteria, including arthritis and back pain27,28.
The poor sensitivity found in our study stands in contrast to the good levels of agreement found by Sangha, et al in the original description of the SCQ20, in which the self-reports of comorbidities were also compared to chart review. The patients in the Sangha study were all inpatients and those in our study outpatients. We hypothesize that the differences may be due, at least in part, to the fact that inpatients may have recently reviewed their medical history and medications with clinical staff at the time of admission and thus have a better recall of their comorbidities compared to our outpatients, who self-administered the comorbidity questionnaire. Again, this underscores that the accuracy of self-reports may depend on the population under study.
A limitation of this study and of chart review studies in general relates to the completeness of medical records. In our study, in and outpatient charts from a single institution were reviewed. Nevertheless, it is possible that, had outpatient charts from other physicians involved in the patients’ care but not available at our institution been reviewed (e.g., general practitioners or medical specialists working in private offices or affiliated with other institutions), more diagnoses reported by the patients could have been confirmed (thereby increasing the positive predictive values). It is possible that diagnoses made on the outside would have been recorded by the treating rheumatologist or another treating physician in our institution, although that is difficult to verify. A second limitation may relate to the way the question used to elicit self-reports was framed. We asked patients to “indicate if you currently have the problem.” Thus, patients with controlled hypertension or cancer in remission may have failed to indicate that they had those conditions, thereby leading to low sensitivity of those self-reports. Finally, we conducted a chart review and extracted physician diagnoses only. It is possible that, had we broadened our data extraction to include investigations, we could have confirmed additional diagnoses (and again, improved the positive predictive values). For example, busy physicians assessing patients with a serious condition such as SSc may fail to mention in their notes that the patient also has OA. Nevertheless, it is possible that, since all patients included in this study were assessed primarily by a rheumatologist, this may have somewhat reduced this problem in so far as diagnoses of OA are concerned. Finally, although our patients had an average education level (45% had some education beyond high school and only 6% had less than 7 years of schooling), educational attainment should be borne in mind in interpreting the results of our study as it could affect the accuracy of self-reports.
In conclusion, we are left with the disquieting observation that the problem of documenting comorbidities in systemic autoimmune diseases remains largely unsolved — self-reports may not capture the truth well for some common, chronic conditions, and chart review is labor intensive, costly, and may fail to provide complete data. Our findings have implications beyond the scope of our own SSc research. Researchers in many other areas, both within and beyond the rheumatology community, routinely collect data on comorbidities using self-reports. They should be aware that self-reports of comorbidities may be prone to considerable error (poor sensitivity from under-reporting and poor positive predictive value because of inaccurate reports) and that adjustment for comorbidities using self-reported diagnoses may be flawed. Methods of assessing comorbidities in observational research and clinical trials have yet to be optimized.
Acknowledgments
CSRG Investigators: J. Pope, London, Ontario; J. Markland, Saskatoon, Saskatchewan; D. Robinson, Winnipeg, Manitoba; N. Jones, Edmonton, Alberta; N. Khalidi, Hamilton, Ontario; P. Docherty, Moncton, New Brunswick; E. Kaminska, Hamilton, Ontario; M. Abu-Hakima, Calgary, Alberta; S. LeClercq, Calgary, Alberta; A. Masetto, Sherbrooke, Quebec; D. Smith, Ottawa, Ontario; E. Sutton, Halifax, Nova Scotia; J-P. Mathieu, Montreal, Quebec; S. Ligier, Montreal, Quebec; M. Fritzler, Advanced Diagnostics Laboratory, Calgary, Alberta.
Footnotes
-
Supported in part by the Canadian Institutes of Health Research, and the Scleroderma Society of Canada, and by educational grants from Actelion Pharmaceuticals and Pfizer Inc. Dr. Hudson is supported by a New Investigator Award from the Canadian Institutes of Health Research and Établissement de Jeunes Chercheurs award from the Fonds de la Recherche en Santé du Québec.
- Accepted for publication February 10, 2009.