Considerations for Evaluating and Recommending Worker Productivity Outcome Measures: An Update from the OMERACT Worker Productivity Group

Special Interest Group (SIG) OMERACT 2018

At OMERACT 14, we presented an update of our work on the first 3 workstreams. Attendees at our SIG included patients (n = 4), clinicians (n = 7), 1 fellow, and others (e.g., methodologists, industry, n = 5). Important questions were discussed with participants during breakout sessions, including:

1. Global measures: Based on the results presented (reliability, cross-cultural differences, construct validity), what would be your preferred global measure and why?

2. Multiitem measure: Based on the context of your research, or your experience as a patient, what do you think are the advantages and drawbacks of using answers that assess both the degree of difficulty and the amount of time with difficulty?

3. PAS/MID: (1) How best to manage MID thresholds and (2) do you agree with the need to report multiple MID/PAS thresholds?

Ethics approval was obtained for individual studies and all patients provided written informed consent [Making It Work trial: University of British Columbia Research Ethics Board (H11-03527); the EULAR-PRO (European League Against Rheumatism — Patient Reported Outcomes) study obtained overall ethical approval from National Research Ethics Service Committee NW–Greater Manchester (12/NW/0172), and from each participating center according to national guidelines].

Global measures

To address the meaning of at-work productivity loss measures from a patient’s perspective in different cultures, we conducted the international EULAR-PRO study to assess presenteeism in patients with inflammatory arthritis (IA) or osteoarthritis. The results of phase I have been published and show fair to excellent test-retest reliability [ICC for Health Productivity Questionnaire (HPQ; question C; 0.59) to WPS-RA (0.78)]¹¹. In-depth cognitive debriefing interviews revealed variation in how participants interpreted some of the constructs among the 5 measures, especially regarding “performance” in the HPQ scale, which was a term used in sport and theatre but not related to work for participants from Romania and Sweden¹². For most participants (∼70%), a recall period of 7 days up until a month would be a good reflection of the effect their health has on work. Phase II is an international observational cohort study (n = 8 countries) to further test psychometric properties. Preliminary results of baseline data on construct validity were presented during the SIG and show moderate to good construct validity (Table 1)¹³. During the break-out session, SIG attendees agreed that a recall period of 1 day was not representative, although they thought a recall period of a month might be too long. Other discussion points included wording of anchors (e.g., normal). Further, participants highlighted the difficulty in answering and interpreting disease-specific scales, because of the complexity of many rheumatic diseases, and preferred a generic scale.

Multiitem measures

How to best measure presenteeism using multiitem scales remains challenging. The WLQ and WALS are frequently used, but participants’ feedback expressed concern about the constructs measured by each instrument. The WLQ measures the amount of time people are limited, but not the extent to which they are limited. This was perceived as a drawback by patients who felt it misses an important part of their experience and by researchers interested in evaluating presenteeism as a health state. In contrast, the WALS measures the extent of limitation but not time. This was a drawback to health economists, because of difficulty assigning cost. To evaluate psychometric properties encompassing both concepts, items from each measure (WALS and WLQ) were offered both time and difficulty response keys (dual answer keys).

Baseline and 6-month data were used from a Canadian randomized controlled trial (Making It Work Program) of an employment intervention including patients with IA (n = 364)^14,15. The psychometric properties of the measures were first evaluated with the 2 answer keys analyzed separately (i.e., without combining results)¹⁶. Answers from the dual answer keys were then combined into a single score, obtained by (1) multiplying or (2) adding scores of difficulty and time answer keys at the item level¹⁷. No significant differences were observed between additive and multiplicative models. High correlation (≥ 0.8) between difficulty and time was found in only 2/12 WALS items and 11/25 WLQ, justifying the need for dual answer keys. High internal consistency (i.e., ≥ 0.7) was found for WALS and all WLQ subscales for both answer keys analyzed separately and combined (except WLQ-Physical Demands)¹⁶. As a priori hypothesized, moderate correlations were observed between original answer keys, or combined scores, of WLQ subscales and WALS with measures assessing similar concepts [WPAI; work instability scale (WIS; congruent validity)]. During the SIG, all agreed that dual answer keys provided additional value. Patient representatives uniformly said they felt that asking both degree and time with difficulty better reflected their experience, and that asking time alone would miss an important concept. The main concern raised was the length and complexity of the questionnaire with both answer keys. Other issues raised included concern about the 2-week recall period and descriptors for time options (considered difficult to answer by patients), and concern about percentage of time attributed to descriptor (e.g., some of the time = 50% of the time).

Thresholds of meaning for worker productivity measures

Thresholds of meaning are benchmarks for scores (e.g., PAS of pain) or change in scores [e.g., minimal threshold for change to be important (MCID)] that aid in the interpretation at an individual patient level. Recently, Copay, et al has demonstrated that there are considerable differences in MCID thresholds depending on the anchor or method¹⁸. At OMERACT in 2018, our focus was on dealing with these differences. As a group, we had reviewed the literature on these attributes and decided on best methods for their determination. In doing so, we emphasized the pivotal role of a meaningful anchor that becomes a gold standard for threshold determination, and the methods used to determine the actual cutoff. We fielded several anchors and provided several analytic approaches to each, allowing us to see the differences in values obtained, which also led to differences in the proportion deemed to be “improved” or “in an acceptable state” (Figure 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Variation in minimal important difference estimates for WALS score depending on anchors and analytical approaches. WALS: Workplace Activity Limitations Scale; DOR: Diagnostic Odds Ratio; P75: 75th percentile.

During our SIG, most of the attendees agreed that we will need to work with a range of MID values. There are also new developments and approaches in reporting results for thresholds, such as cumulative distribution function¹⁹. The various thresholds for MCID are highlighted with a vertical line on the same graph and demonstrate not only the proportion responding, but whether various MCID values would lead to different interpretations of the relative gains. Another approach discussed was the cumulative proportion responders analysis graph²⁰, which plots proportion responders (as defined by having exceeded the MCID) against magnitude of change with 1 line for each arm in a trial. For clinical trials, this allows more transparent interpretation of the difference between arms. In MCID development work, a plot for each MCID value in a cohort would allow us to see whether different MCID thresholds had a large or small difference in the proportion classified as improved. The breakout groups agreed that these reporting approaches could improve the management of multiple MCID values. They will be forwarded to the Technical Advisory Group of OMERACT for consideration.

Key points resulting from the SIG:

• A dual scale, measuring both time and difficulty, better identifies patient’s experience, but the main drawback is the length and complexity of such a scale.

• There is no perfect global scale, but a generic scale with a recall > 1 day and < 1 month is preferred.

• Development of reporting approaches is key to improve management of multiple MCID values.