Patient reported health status is considered a key element in the assessment of rheumatic diseases, both in research and in clinical practice¹ and is part of recommended core measures for clinical studies.^2–4 A patient reported rheumatoid arthritis disease activity index (RADAI) has also been successfully developed.⁵

The OMERACT meetings in 2002 and 2004 included patient representatives in the evaluation of outcomes, under the umbrella of “patient perspective in outcome research.”^6–9 The patients had many suggestions about the research agenda in this area. One was more frequent monitoring of health status to capture the fluctuations in health over time. Many patients complete diaries, but access to and analyses of such data may be difficult, because the data often are recorded without standardisation and need to be entered into a computer for further analysis. It was recognised that electronic recording of health status on the internet or on a personal digital assistant (PDA) instead of using the traditional paper/pencil format could provide opportunities for daily or even more frequent assessments.⁶ It has been emphasised that the electronic approach to data collection provides opportunities for future clinical practice and research within many different medical areas.^10–13

At OMERACT-7 in 2004 it was agreed that information technology data collection techniques for outcome assessment should be adequately validated.⁹ A few publications have appeared previously in rheumatology journals in this area—for example, focusing on the validity of the American College of Rheumatology (ACR) patient assessment questionnaire^14,15 and on a computerised version of the short form 36 item health assessment questionnaire (SF-36) in patients with rheumatic diseases.^16,17 The objective of the present study was to compare self reported health status measures, including the RADAI, in a PDA version and a paper/pencil version regarding test–retest reliability, score agreement, and feasibility.

METHODS

Patients were recruited to the study from a county rheumatoid arthritis register¹⁸ if they were between 50 and 70 years of age. A letter informing them about the project was sent to several patients who were randomly drawn from the register. Patients were included if they were willing to participate and had a stable disease with no change in drug treatment and no surgical procedures during the previous four weeks. The recent disease history was determined from the patient and from the hospital records.

The patients were examined on two occasions (T1 and T2, five to seven days apart) with self reported health status measures recorded in two different ways: in a conventional paper/pencil format and on a PDA (HP iPAQ, model h5450). The individual patients were gathered into four groups who met in the hospital in the late afternoon to participate in the project. Patients in two of the groups started with the paper version on both occasions, the other two started with the PDA version. The instruments included joint pain, fatigue, and patient global evaluation of their disease on visual analogue scales (VAS), RADAI,⁵ modified health assessment questionnaire (MHAQ),¹⁹ and SF-36.²⁰ A trained study nurse carried out joint counts on both occasions, and acute phase reactants were also examined to verify stability of the disease activity.

The patients were asked to rate the satisfaction with each method (PDA v paper/pencil) on a 100 mm VAS.

Feasibility was also assessed by the time needed to complete the self reported measures and by a final question asking for the preferred method. We also recorded whether patients were able actually to use the PDA.

Analysis of data

Recorded data in the PDA were transferred through a wireless local area network to a computer, and then imported as a text file into SPSS for analyses. The paper data were entered manually into the database. Test–retest reliability was examined by the Bland–Altman approach,²¹ computing the smallest detectable difference (SDD)—that is, 1.96*SD of the difference between the scores, as well as the limits of agreement,²¹ by the coefficient of variation (CV%), and by intraclass correlation coefficients (ICC). We used a two way mixed effects model to calculate the single score ICC. The reliability of the PDA and paper versions was compared by an analysis of variance (ANOVA) on the differences. Relations between the scores were examined by Pearson correlation coefficients and agreement by the computed differences between the scores with a 95% confidence interval (CI) as well as a paired t test. The agreement between the scores were also illustrated by Bland–Altman plots.²¹ The level of statistical significance was set to p<0.05.

The study was approved by the regional ethics committee.

RESULTS

Thirty patients were included after giving their informed consent (table 1)

The SDDs, limits of agreement, CV%, and ICCs are shown in table 2 (data for SF-36 role scales and social functioning are not included). The SDDs were rather high, but similar for paper and PDA. The ICC varied between 0.69 and 0.93 for the paper versions and between 0.66 and 0.94 for the PDA versions. The results of analysis of variance confirmed that the reliability was similar with the two methods (p>0.10 for all measures, data not shown). Measures of reliability were of the same magnitude even when replacing one method by the other at T1 and T2 (for example, examining scores by paper at T1 and PDA at T2).

The scores obtained with the two methods correlated strongly with each other (r values between 0.79 and 0.97 at T1 and 0.82 to 0.98 at T2) (table 3). Higher VAS scores were seen with the paper than with the PDA versions in four of the six measures (p = 0.03 to 0.05), but the statistical significance disappeared after Bonferroni correction for the number of tests,. No differences between the methods were observed for the scale scores (MHAQ, RADAI, and SF-36) except for a difference for SF-36 bodily pain at T1 and mental at T2, with more severe scores for the PDA than for the paper version (table 3). The agreement between scores obtained by the PDA and paper versions is also illustrated by Bland–Altman plots for RADAI and pain VAS at T1 in fig 1A and 1B.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

 Agreement between scores obtained by the PDA and paper versions illustrated by Bland–Altman plots for rheumatoid arthritis disease activity index (RADAI) (panel A) and pain visual analogue score (VAS) (panel B) at T1.

All patients managed to use the PDA. The mean (SD) satisfaction score at T1 with paper was 67.9 (18.1) and with PDA, 60.4 (11.4) (p = 0.29). Mean (SD) time to complete was 24.9 (27.0) and 30.5 (16.0) minutes, respectively (p = 0.11). Similar results were seen at T2. At the end, 20.7% reported preference for the paper version, 65.5% for the PDA, and 13.8% had no preference.

DISCUSSION

This study showed that widely used health status measures performed similarly in a format for PDA and in the traditional paper/pencil format. The recent OMERACT-7 meeting recommended that electronic data collection techniques for outcome assessment should be adequately validated.⁹ It is possible that electronic and paper versions of questionnaires may provide different results, taking into account that patients with a PDA are presented with one question at a time, and then have to proceed to the next screen to see the next question. Thus there are 36 screens for SF-36, 13 for RADAI, and one screen for each VAS. In comparison, in the paper based version all these questionnaires were presented on eight pages.

This study focused on test–retest reliability, but additional validation with a broader focus on psychometric properties, including responsiveness, is also required. Previous studies in rheumatic patients have shown that computerised versions of SF-36 perform similarly to the traditional paper/pencil version,^16,17 in accordance with findings from computerised versions of health related quality of life instruments in other diseases.^22–24 The ACR patient assessment questionnaire also performed similarly in paper/pencil and computer versions.^14,15 Collection of patient reported health status on PDA has also been successfully validated in experiments focusing on VAS²⁵ and in patients with acute^11,26 and chronic pain,²⁷ orthopaedic conditions,²⁸ gastro-oesophageal reflux,²⁹ and Parkinson’s disease.³⁰ Scores that were obtained with PDA and paper in the present study were also similar (table 3), but the Bland–Altman plots indicated that some individuals had major differences between their scores (fig 1A and 1B).

The test–retest reliability of patient reported measures was satisfactory when considering the ICCs. The 95% SDDs provide clinically useful information, as they represent the cut off values that have to be exceeded if a clinician wants to be 95% confident that a change reflects a significant improvement or deterioration. These values were 0.16 with paper and 0.20 with PDA for MHAQ, 1.11. and 1.24 for RADAI, and 14.8 and 12.7 for SF-36 physical, respectively (table 2). The value for MHAQ was close to the change that has been considered clinically important with the health assessment questionnaire (HAQ).³¹ We also repeated the joint counts and the acute phase reactants at T2. For comparison the SDD of the 28 swollen joint count (28-SJC) was 2.8, for the 28 tender joint count (28-TJC) it was 3.48, for investigators global it was 8.0, and for ESR/C reactive protein it was 5.7/4.8 (data not shown). The magnitude of the measurement errors is important when clinical decisions are based on changes in scores in individual patients. These measurement errors also suggest that frequent monitoring may be relevant for clinical decision making in individual patients, as repeated measures may provide more reliable information than, for example, single assessments at scheduled clinic visits. One feasible approach to obtaining such frequent data is electronic recording of patient reported data^27,30 with wireless transferral of data directly into the hospital computer system.

Recall of a previous recording of health may influence the second recording. We assume that this problem is reduced when large numbers of instruments are completed as in this study. The completion of the instruments should ideally have been done in a randomised crossover design. The patients in this study were for practical reasons examined in four groups of six to eight individuals, and all patients within one group completed all instruments in the same sequence. However, two of the groups started with PDA and the other two started with the paper/pencil version.

Individual patients may differ with regard to acceptance of new technology. We chose to carry out this study in a typical age span of rheumatoid arthritis—that is, 50 to 70 years—and not in younger patients where acceptance could be expected to be high. This study was not powered to explore the performance of the electronic version in subgroups of patients based on sex, level of education, or age. The impression from the investigators was that no particular subgroup preferred one method to the other, and studies in other patient groups have not revealed any major problems in subgroups based on computer literacy, educational level, age, sex, or race.^14,22,23

We had expected that feasibility would be higher with paper than with PDA, taking into account that we recruited regular patients without any particular bias in the direction of new technology. Time to complete was numerically lower with paper than with PDA. We also had experiences during the study that may lead to improved feasibility. For example, some dissatisfaction was related to slowness in the electronic wireless transferral of data from the PDA to the computer.

Feasibility for researchers obviously favours electronic recording, as the data may be fed directly into the computer system without any data entry procedures, which may prevent incorrect data being entered and reduce costs. Compliance with data recording might be improved because patients are prompted to answer questions at the right moment—that is, they have to complete one screen before they are allowed to proceed to the next.^11,26 Furthermore, the time and date of the completion are also recorded.

Electronic versions of patient reported health status have major potential for the future. With increasing demands for documentation of results of expensive drugs and other interventions in health care, patients may do their self assessments at home in the near future. Different types of technology can be used. The advantage of PDA is cost and flexibility (the device can be carried and used everywhere). However, access to home based computers or laptops will increase in the future, and computers with touch screens for data recording in the hospital represents another opportunity. Williams et al¹⁴ have described the positive and cost saving experiences with a series of assessments from questionnaire in a setting of 1062 patient visits. Data collected by computers can be accessed immediately on the clinician’s computer screen if the data system includes software that can compute scores (for example, MHAQ, SF-36, RADAI) from the data that have been entered. Mistakes during data entry can be prevented by a warning if values outside predefined ranges are entered, and costs related to data entry from paper questionnaires are saved. Data entered in patients’ homes can also be transferred to the hospital network through a cell phone network (in the case of PDA) or through the internet (in the case of a personal computer).

The potential is probably also large in clinical trials. Frequent assessments may be mandatory for some interventions—for example, treatment of acute pain—and responsiveness may be improved if repeated measures are available instead of a single measure from one time point. Validation of responsiveness was not part of this study but it is one of the next steps in the additional validation of health status reported on a PDA in patients with rheumatoid arthritis.

Are there any potential disadvantages? Some patients may have barriers to the use of new technology,¹⁴ even though this was not experienced in the present investigation and has not been a problem in other settings.^22,23 Security systems with firewalls around the hospital server are mandatory to secure confidentiality and privacy. Further, patients with severe or worsening hand problems may not be able to use the small stylus to enter data in a PDA. This may cause selective drop out from the study of patients with worsening hand function. In this project we were particularly aware of this potential problem, as also one of the members of the research group (ØD) is an occupational therapist. However, our experiences indicate that almost all patients can use PDAs, and some patients with finger deformities replace the stylus with the fingertip on the touch screen of the PDA or a thicker pen. We did not observed problems with visibility of the screen, and all patients managed to use the PDA.

The performance of paper based versions of self reported health status measures and electronic versions, using an inexpensive PDA, was similar in this study. In our opinion, both regular computers and PDA have future potential for monitoring and assessing disease. Advantages of PDA are the size and availability at virtually any time and place, provided there is access to power to charge the batteries. Our results encourage validation of electronic recording of self reported health status in different clinical and research settings in rheumatology, and especially further research on the responsiveness and the daily recording of health status in the patients’ homes.