Original Article
Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function

https://doi.org/10.1016/j.jclinepi.2008.01.005Get rights and content

Abstract

Objective

Assess practicality of using a computerized adaptive test (CAT) in routine clinical practice, perform a psychometric evaluation of content range coverage and test precision, and assess known group construct validity, sensitivity to change and responsiveness of knee CAT functional status (FS) measures.

Study Design and Setting

Secondary analysis of retrospective intake and discharge rehabilitation FS data collected in a prospective cohort study. Data represented a convenience sample of 21,896 patients with knee impairments receiving outpatient physical therapy in 291 clinics in 30 U.S. states (2005–2007).

Results

The CAT used an average of seven items to produce precise estimates of FS that adequately covered the content range with negligible floor and ceiling effects. Test information functions and standard errors supported FS measure precision. FS measures discriminated patients by age, symptom acuity, surgical history, condition complexity, and prior exercise history in clinically logical ways. Seventy-two percent of patients obtained discharge FS measures ≥ minimal detectable change (95% confidence interval). Change of 9 FS units (0–100 scale) represented minimal clinically important improvement, which 67% of patients obtained.

Conclusion

The knee CAT was efficient and produced precise, valid, and responsive measures of FS for patients receiving therapy for knee impairments and functioned well in routine clinical application.

Introduction

Measurement of patient-reported outcomes (PRO) in health care is evolving with psychometricians capitalizing on the administrative efficiency and measure precision of computerized adaptive tests (CAT) [1], [2], [3]. The primary advantages of CAT administrations [4], [5], [6] are that the adaptive tests tailor PRO item administration, so each patient responds to items whose location, that is, item's position on the latent continuum [7], [8], or difficulty approximates the patient's ability estimates, and the number of items administered is minimized [8], [9], [10], [11]. In this era of clinical efficiency that is increasingly influenced by the need for the collection of PROs that may influence payment policy [12], [13], [14], measure precision and efficiency are important.

CAT has its origins in mental [15], educational [16], and military [5] testing. Recently, CAT applications have been introduced in health care [3], [17]. As described by Jette et al. [18], application of CAT in health care have been recommended for nearly a decade [19], [20], [21], but only recently have CAT been used in data simulations [22], [23], [24], [25], [26], [27], research demonstrations [24], [28], [29], and prospective data collections [18], [30], [31]. Although development of CAT in health care in general is growing [1], [2], [3], [10], [32], development and use of CAT in outpatient rehabilitation specifically [18], [25], [26], [27], [30], [33], [34] is rapidly advancing. Several factors positively influence that growth. Therapists commonly measure PRO because national associations recommend outcomes collection and analysis [35]. Outcomes measures are common in outpatient rehabilitation [35]. Most patients have functional limitations that improve with therapy and can be measured [18], [36], [37]. Payers expect justification for treatment [38].

The current study builds on previous work where we developed and simulated body part specific CAT [25], [26], [27] using retrospective data analyses. Here, we evaluated the practical and psychometric adequacy of a CAT used to assess functional status (FS) of patients with knee impairments seeking rehabilitation in outpatient therapy clinics participating with Focus On Therapeutic Outcomes, Inc. (FOTO), an international medical rehabilitation outcomes database management company [39], [40]. As previously described [26], for the purpose of this study of patients with knee impairments, the latent trait of interest is lower extremity FS, which we operationally define as the patient's perception of their ability to perform functional tasks described in the FS items. Therefore, FS represents the level of ability of the patient, which represents the patient's location on the underlying latent trait of FS [7], [8]. The item bank for the knee CAT [26] was developed using items from the Lower Extremity Functional Scale (LEFS) [41], a scale with strong psychometrics [41], [42], [43], [44] and broad clinical and research acceptance [45]. FS, as assessed using LEFS items, represents the “activity” dimension of the World Health Organization's International Classification of Functioning, Disability and Health [46].

Our purposes were to (1) assess the practicality of using a CAT in routine clinical practice; (2) perform a psychometric evaluation of content range coverage and test precision; and (3) assess known group construct validity, sensitivity to change, and responsiveness of the FS measures estimated using the knee CAT. Analyses will be used to determine strengths and weaknesses of the CAT and help direct future research designed to improve the CAT's practicality and psychometric properties.

Section snippets

Design and setting

We conducted a secondary analysis of retrospective data collected from patients with knee impairments prior to and at conclusion of rehabilitation. Focus On Therapeutic Outcomes, Inc. Institutional Review Board for the Protection of Human Subjects approved the project.

Subjects

Patients with knee impairments, who represent a sample of convenience, were analyzed (Table 1). Identification of medical or surgical diagnoses was optional in the data collection (described below), but of the patients with

Efficiency

Scale distributions, number of treatment visits, and treatment duration are displayed in Table 2. Patients who had both intake and discharge data, compared to patients with just intake data, were older (52 years [SD = 17] vs. 49 years [SD = 16], t = 13.6, df = 18,807, P < 0.001), had more chronic symptoms (chi-square = 19.8, df = 2, P < 0.001), had more comorbidities (chi-square = 9.1, df = 3, P = 0.029), took more medication at intake for their condition (chi-square = 30.9, df = 1, P < 0.001), were more likely to be

Discussion

CAT are common in standardized testing for licensure, certification, and admissions tests [4], [6] but have only recently begun to be used to collect routine clinical data in busy outpatient rehabilitation clinics in the United States [18], [31] and Israel [30]. Results of the current study support the conclusions of Jette et al. [18] that the potential is good for CAT to efficiently generate estimates of FS that are precise, valid, and sensitive to change. These conclusions are encouraging for

Acknowledgments

The authors would like to thank Karon F. Cook, PhD for her insightful comments regarding statistical analyses, results, and manuscript edits.

References (92)

  • J.G. Wright et al.

    A comparison of different indices of responsiveness

    J Clin Epidemiol

    (1997)
  • B. Kirshner et al.

    A methodological framework for assessing health indices

    J Chronic Dis

    (1985)
  • D.E. Beaton et al.

    Evaluating changes in health status: reliability and responsiveness of five generic health status measures in workers with musculoskeletal disorders

    J Clin Epidemiol

    (1997)
  • R.D. Crosby et al.

    Defining clinically meaningful change in health-related quality of life

    J Clin Epidemiol

    (2003)
  • R. Jaeschke et al.

    Measurement of health status. Ascertaining the minimal clinically important difference

    Control Clin Trials

    (1989)
  • R.A. Deyo et al.

    Assessing the responsiveness of functional scales to clinical change: an analogy to diagnostic test performance

    J Chronic Dis

    (1986)
  • D.L. Hart et al.

    Development of an index of physical functional health status in rehabilitation

    Arch Phys Med Rehabil

    (2002)
  • H. Fliege et al.

    Development of a computer-adaptive test for depression (D-CAT)

    Qual Life Res

    (2005)
  • J.E. Ware et al.

    Practical implications of item response theory and computerized adaptive testing: a brief summary of ongoing studies of widely used headache impact scales

    Med Care

    (2000)
  • R.K. Hambleton et al.

    Fundamentals of item response theory

    (1991)
  • R.D. Hays et al.

    Item response theory and health outcomes measurement in the 21st century

    Med Care

    (2000)
  • J.B. Bjorner et al.

    The feasibility of applying item response theory to measures of migraine impact: a re-analysis of three clinical studies

    Qual Life Res

    (2003)
  • D. Cella et al.

    The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years

    Med Care

    (2007)
  • R.K. Hambleton

    Emergence of item response modeling in instrument development and data analysis

    Med Care

    (2000)
  • Institute of Medicine

    Crossing the quality chasm: a new health system for the 21st century

    (2001)
  • Institute of Medicine

    Rewarding provider performance: aligning incentives in medicare

    (2006)
  • M.E. Porter et al.

    Redefining health care. Creating value-based competition on results

    (2006)
  • F.M. Lord et al.

    Statistical theories of mental test scores

    (1968)
  • F. Lord

    Some test theory for tailored testing

  • J.E. Ware et al.

    Applications of computerized adaptive testing (CAT) to the assessment of headache impact

    Qual Life Res

    (2003)
  • A.M. Jette et al.

    Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings

    Phys Ther

    (2007)
  • C.A. McHorney

    Generic health measurement: past accomplishments and a measurement paradigm for the 21st century

    Ann Intern Med

    (1997)
  • D.L. Patrick et al.

    Convening health outcomes methodologists

    Med Care

    (2000)
  • D.A. Revicki et al.

    Health status assessment for the twenty-first century: item response theory, item banking and computer adaptive testing

    Qual Life Res

    (1997)
  • W. Gardner et al.

    Multidimensional adaptive testing for mental health problems in primary care

    Med Care

    (2002)
  • S.M. Haley et al.

    Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme

    Clin Rehabil

    (2006)
  • J.E. Ware et al.

    Item response theory in computer adaptive testing: implications for outcomes measurement in rehabilitation

    Rehabil Psychol

    (2005)
  • D. Deutscher et al.

    Implementing an integrated electronic outcomes and electronic health record process to create a foundation for clinical practice improvement

    Phys Ther

    (2008)
  • D.L. Hart et al.

    Pay-for-performance for physical therapy and occupational therapy: medicare part B services. Grant #18-P-93066/9–01

    Health & Human Services/Centers for Medicare & Medicaid Services

    (2006)
  • D.N. Ader

    Developing the Patient-Reported Outcomes Measurement Information System (PROMIS)

    Med Care

    (2007)
  • S.M. Haley et al.

    Activity outcome measurement for postacute care

    Med Care

    (2004)
  • American Physical Therapy Association

    Guide to physical therapist practice

    Phys Ther

    (2001)
  • L. Resnik et al.

    Using clinical outcomes to identify expert physical therapists

    Phys Ther

    (2003)
  • Cited by (0)

    View full text