Worker Productivity Outcome Measures: OMERACT Filter Evidence and Agenda for Future Research

Progress Leading up to OMERACT 11

Leading up to OMERACT 11, our focus has been the presenteeism component of worker productivity and the appraisal of relevant instruments such as patient-oriented outcome measures for use in rheumatology research. With growing collaborations within our working group and a more focused set of measures under consideration for endorsement, we have decided that our process would benefit from an organization into multiple work-streams to pursue several related topics. Currently, we have 4 active work-streams (Table 1): the first is focused on evaluating and comparing global approaches to measure at-work limitations/productivity loss (e.g., single-item measures); the second is focused on multi-item approaches to measure at-work limitations/productivity loss (e.g., summative scales); the third is focused on contextual factors of worker productivity; and finally, the fourth work-stream is focused on interpretability of worker productivity outcomes, which entails examination of scale cut-scores associated with patient acceptable state (PAS)^10,11 and score changes that would represent a minimally important difference (MID)^12,13. Progress specifically pertaining to each of these 4 work-stream and current research agendas will be discussed in the following sections.

In preparation for OMERACT 11, our working group engaged in several initiatives to help address gaps in OMERACT filter evidence among contending presenteeism measures, particularly in the area of discrimination (i.e., reliability, responsiveness). We updated our literature review of available psychometric evidence, including randomized controlled trials (RCT, pharmacological or nonpharmacological) in arthritis or musculoskeletal populations where worker productivity measures have been applied as a study outcome. Results from these trials informed whether the presenteeism component of an instrument can afford that fine level of discrimination required to evaluate change scores in 1 treatment arm over another treatment/control arm (i.e., between-group differences of within-group change). Details of these RCT are summarized in Supplementary Table 1 available online at jrheum.org. In addition, through funding support from a Discovery Advancement Project research grant from the Canadian Arthritis Network (CAN), we initiated an international Web-based survey recruiting workers with arthritis to examine the test-retest reliability (reproducibility of scores) and score interpretability (PAS/MID estimation) of candidate measures of at-work limitations/productivity. Preliminary results on the test-retest reliability of presenteeism measures revealed a moderate-to-high range of intra-class correlation coefficients (ICC_2,1)¹⁴, suggesting good to very good agreement between test-retest scores (Table 2). Tests of agreement were conducted among participants who were considered stable in their work ability over a 2-week period, based on having indicated “no change” on an 11-point external anchor (0 = much worse, 5 = no change, 10 = much improved) assessed at 2-week followup.

Objectives and Proceedings of the OMERACT 11 Workshop

At the OMERACT 11 Workshop, our overall aim was to appraise whether various candidate global measures of at-work limitations/productivity currently meet the requirements of the OMERACT filter, through a formal plenary vote. At the 2-h workshop, we initially provided background presentations to review our working group’s progress from previous OMERACT meetings, and a current update of OMERACT filter evidence among our candidate instruments. We then invited 2 of our consumer partners to speak and shed light on the relevance and significance of employment issues from their own personal experience. This was followed by 8 individual breakout sessions led by our working group members. During the first part of all individual breakout sessions, we focused on a common topic — a discussion of the content and available psychometric evidence for the various global measures of at-work limitations/productivity — to allow participants to compare and contrast the measures in preparation of the plenary vote. The second part of the breakout sessions discussed unique topics that corresponded to our other work-streams: 3 of the individual breakout sessions discussed global measures of presenteeism in more detail, 2 of the sessions discussed the current evidence on multi-item measures and score interpretability issues, and the 3 remaining sessions discussed contextual factors of worker productivity. Finally, at the end of the individual breakout sessions, all conference participants reconvened for a plenary voting session where a recap of OMERACT filter evidence was presented, and then specific questions were posed to participants to formally evaluate whether our candidate presenteeism global measures currently meets the requirements of the OMERACT filter.

Work-stream 1

Global measures are designed to ask about “on-the-job” problems or at-work productivity (loss) using a single or just a few items. Typically, these items ask fairly broad questions and are intended to provide an overall appraisal or gestalt of perceived limitations or productivity at work. Although some global measures can be found in larger, modular-type questionnaires that also concurrently assess other aspects of work productivity (such as work absenteeism or loss of household productivity), it is important to be able to demonstrate that the presenteeism component, in and of itself, is meeting the requirements of the OMERACT filter. Moreover, this aspect of worker productivity is often reported as a discrete outcome in studies, as many researchers are interested in understanding whether interventions can influence both work absenteeism and presenteeism. Currently, 5 global measures of at-work limitations/productivity are under consideration (Table 3), 3 of which are based on voting from OMERACT 9. These include the presenteeism component of the Rheumatoid Arthritis-specific Work Productivity Survey (WPS-RA)¹⁵, Work Productivity and Activity Impairment questionnaire (WPAI)¹⁶, and the World Health Organization (WHO) Health and Work Performance Questionnaire (HPQ)^17,18. In addition, 2 other measures were added to address the need for measures of work ability/difficulty: the Quantity and Quality Method (QQ)¹⁹, which was favored by experts; and lastly, the central (first) item of the Work Ability Index (WAI)²⁰, which has emerged in recent years as a popular work outcome measure, particularly within the European community. All 5 global presenteeism measures use a numeric rating scale (Table 3), and all except the WAI are considered to have potential for use in economic evaluations.

At individual breakout sessions at our OMERACT 11 Workshop, we reviewed the content of the global measures and highlighted 4 key aspects where these measures exhibit some important differences. The first is the intended measurement concept, which is reflected by the different terms used to describe work issues. Global items from the WPAI and WPS-RA assess work “productivity,” the HPQ evaluates work “performance,” the QQ method focuses on “how much” (i.e., quantity) and “quality” of the work performed, while the WAI asks for a rating of work “ability,” offering a perspective that is distinct from productivity. The second element is the length of the recall period, which varies from “current” to “last month” among our 5 candidate instruments. A third element is whether there is disease attribution; that is, whether the item(s) asks respondents to consider work problems insofar as they are deemed to be directly attributed to a specific or non-specific health problem. Only the WPAI (specific health problem version) and WPS-RA are disease-specific, while all other candidates are formulated without any disease/health attributions. Fourth and finally, the measures also differ in terms of comparative referencing. This concerns whether respondents are asked to provide a rating against a “reference” situation/worker. Only the QQ method and global items from the WAI and HPQ apply a type of comparative referencing. The QQ method asks respondents to compare against “a normal work-day,” while WAI asks respondents to provide a rating “in relation to lifetime best.” As per developer recommendation²¹, the presenteeism global from HPQ can be scored 2 ways: as a ratio against another item that assesses the usual performance of most workers (“relative presenteeism”), or simply as a standalone single-item (i.e., no comparative referencing).

OMERACT Filter Evidence

Summary ratings for the various aspects of the OMERACT filter for each of the 5 candidate global measures of at-work limitations/productivity in arthritis or musculoskeletal populations are provided in Table 4. These ratings reflect an up-to-date synthesis of available and relevant measurement evidence for individual measures (full details in Supplementary Table 2 available online at jrheum.org), which were agreed upon among working group members. Overall, our current appraisal suggests that all 5 presenteeism global measures satisfy both truth and feasibility criteria, and it is in the discrimination criterion where there are some differences. Currently, only the presenteeism global measures from the WPAI and WPS-RA are considered to have fully satisfied the discrimination criterion, given the availability of supporting evidence for test-retest reliability, responsiveness, RCT (demonstrated differences between intervention groups where efficacy would be expected), and preliminary findings on PAS/MID estimates presented at OMERACT 11 (unpublished results). On the other hand, evidence for responsiveness currently remains lacking for the HPQ global and QQ method, while supporting evidence of between-group discrimination from RCT is also not yet available for presenteeism global measures from the HPQ or WAI.

Plenary Voting at the OMERACT 11 Workshop

At the plenary voting session, we asked conference participants to vote on the following question for each of the 4 presenteeism global measures from the WPAI, WPS-RA, HPQ, and the QQ method: Do you think this measure has enough evidence to pass the OMERACT filter (response options: yes, no, don’t know)? Both the WPAI (yes = 72%) and WPS-RA (yes = 71%) met the minimum 70% threshold required for endorsement. Both the HPQ (yes = 8%) and QQ method (yes = 13%) were well below the level of support required for endorsement, which was anticipated considering remaining gaps in discrimination evidence from our reviews. Because the WAI was only recently identified for OMERACT consideration and because it assesses a different perspective (i.e., work ability), for this measure, we specifically asked: Do you agree that the (presenteeism global measure from) WAI should be considered as a candidate measure of ability/difficulty at work for potential OMERACT endorsement, pending demonstration of filter evidence (response options; yes, no, don’t know)? Among conference participants, 70% supported this global measure as a candidate instrument; therefore it met the level of support required. The full voting results are presented in Table 5, and it should be noted that calculations of level of endorsement were based on proportions of “yes” or “no” votes and excluded “don’t know” votes, as per OMERACT guidelines. Among the questions posed, we found that proportions of “don’t know” votes were generally small (6–17% of all votes).

Feedback from breakout sessions provided directions for future research for this work-stream. Conference participants suggested that the availability of evidence that compared productivity-oriented presenteeism measures to objective indicators of on-the-job productivity (e.g., quantifying actual work output over a defined period) could provide more definitive and convincing evidence of construct validity. Some concerns were raised about appraising only the presenteeism component of full questionnaires, although participants also understood the need to validate components of worker productivity independently if these are to be presented as discrete outcomes in studies and trials. Some participants remarked on the complexity of the concept of worker productivity, and the potential difficulties of answering broader global items without a better understanding of what is specifically implied by the terms “work productivity” or “work performance” or “work ability.” To this end, it was also suggested that clear definitions of such items would be helpful for the respondent. This was thought to be an important contrast against items found in summative multi-item scales, which are typically more specific and thus may be advantageous in terms of facilitating more consistent interpretations of content across respondents. Finally, the need for more head-to-head comparison of candidate global measures to gain further insights into the relative strengths and limitations of individual measures was also suggested.

Research Agenda for Work-stream 1

The research agenda for this work-stream will focus on consolidating evidence of test-retest reliability among our candidate presenteeism global measures through our ongoing project funded by the Canadian Arthritis Network. We are also poised to complete a research project funded by the European League Against Rheumatism (EULAR), in which the comparability of the 5 candidate presenteeism global measures are investigated. The first phase of the EULAR-funded project will involve cognitive interviewing with patients with arthritis to explore their interpretation and understanding of the 5 global measures, with specific focus on the aforementioned 4 key elements (e.g., intended measurement concept, recall period) where important differences exist among the tools. Patients to be recruited for this project will represent workers with different arthritic/rheumatic conditions [e.g., rheumatoid arthritis (RA), osteoarthritis (OA), ankylosing spondylitis] from various occupational sectors (e.g., manual vs non-manual jobs) and geographic regions in Europe and Canada to explore the generalizability of findings. Results from the first phase will be applied to the second phase of the project, which will investigate the ideal recall period and construct validity of the candidate global presenteeism measures.

Work-stream 2

Multi-item measures are designed to assess at-work limitations/productivity through a series of summative items. These are the traditional “psychometric scales” that aim to measure an underlying unobserved (or latent) construct(s). At the OMERACT 9 plenary voting session, 3 multi-item measures received support for continued consideration: the 12-item Workplace Activity Limitations Scale (WALS)⁴, 23-item Work Instability Scale for Rheumatoid Arthritis (RA-WIS)²², and the family of Work Limitations Questionnaire (WLQ-25, WLQ-16, WLQ-8, Work Role Functioning-26/27)^23,24, which uniquely can be applied for economic evaluations. Given our focus on instruments specifically aimed at measuring either “worker productivity” or “worker ability/difficulty,” the RA-WIS is no longer considered a candidate outcome measure because its intended concept of work instability represents a different work concept. However, given its rigorous scale development process²², strong measurement properties in both RA and OA^9,22,25,26, and ability to predict future adverse work outcomes²⁷, we believe the RA-WIS has a unique role as a promising prognostic tool for use in arthritis/rheumatic conditions, and may also be an appropriate outcome in studies where perceived risk of job loss (precariousness of employment) is a primary target of interest.

Multi-item presenteeism measures were a topic of discussion at 2 individual breakout sessions at the OMERACT 11 Workshop. As was the case for global measures, we similarly recognized that the discrimination criterion remains an area with some gaps, in particular, the paucity of RCT using either the WALS or the WLQ-25 that provides evidence of between-group discrimination for these instruments (Table 6). Currently, we have found 5 published or ongoing RCT that used the WLQ-25 (or a variation of the measure), and only 1 has shown a significant difference between treatment arms²⁸. However, this study did not directly measure WLQ-25 scores; instead, scores were imputed from other outcome measures assessed in the trial. Thus, evidence from this study may have limitations in terms of informing the discrimination of the WLQ-25. To our knowledge, no published RCT applying the WALS are available to date, although other types of discrimination evidence (e.g., responsiveness) have been demonstrated from previous psychometric studies (see Appendix 1 and 2).

Direct Comparability of Multi-item Measures

The direct comparability of these measures is a topic of interest, and head-to-head studies using a controlled sample can offer a useful approach to assess the relative strengths and limitations of candidate measures. One perspective of particular interest for direct comparison is the content validity of multi-item measures from patients’ viewpoints, because the WALS and WLQ-25 quantify work problems from differing perspectives. The WALS assesses the degree of workplace activity limitations, while the WLQ-25 is aimed at quantifying the proportion of time a worker experiences work limitations. In a recent study, we directly compared the “sensibility” (includes content validity, feasibility, understandability, suitability of response options) of 5 multi-item measures (including these 2 contenders) from the perspective of workers with either OA or RA²⁹. Overall, both the WALS and WLQ-25 performed well, providing further support for truth (face/content validity) and feasibility aspects of the OMERACT filter. In the study’s final appraisal, patients were asked to provide a forced-choice selection from among the 5 measures that they preferred overall. The WALS (32.6% support) and WLQ-25 (30.0% support) were ranked first and second, respectively.

Physical Demands Subscale of the WLQ-25

The WLQ-25 has 4 subscales — time management, physical demands, mental-interpersonal demands, and output demands. While the same set of response keys is provided for all 25 items (none of the time to all of the time), in the physical demands subscale the instruction uniquely asks respondents to quantify the amount of time “without difficulty,” whereas the other 3 subscales ask respondents to quantify the amount of time “with difficulty.” Feedback from patient respondents has indicated that they found the reversal of wording in the physical demands subscale to be difficult. Respondents missing this subtle but critical change in instruction could represent a potential source of measurement error, which may explain some of the lower-than-expected correlations between WLQ-25 subscales observed in several studies (i.e., r < 0.4)^9,30,31. This has led us to adapt this subscale so that there are consistent instructions across the 4 subscales, and preliminary results from our ongoing CAN-funded study have shown that the intersubscale correlations are behaving in a manner thought to be more consistent with conceptual expectations (r = 0.71-0.87). Because the WLQ-25 is a copyrighted instrument (Glaxo Wellcome, Drs. Debra Lerner and Benjamin C. Amick III), we are currently referring to this modified measure as the “adapted WLQ-25.” We are also currently engaging developers about the acceptability of our proposed adaptation for this highly promising measure.

Research Agenda for Work-stream 2

The future research agenda for the multi-item work-stream involves several objectives. First, as with the global measures, we will continue to update the literature for OMERACT filter evidence, with emphasis on emerging discrimination evidence. Second, we will consolidate evidence of the test-retest reliability for our multi-item candidate measures through our CAN-funded project. Third, we will also explore a novel approach to measure worker productivity, which involves items having dual response keys, allowing concurrent assessments of frequency of difficulty (like the WLQ-25) and amount of difficulty (like the WALS). Both perspectives have been shown to have relevance in the quantification of on-the-job problems, and a combined scale might afford increased efficiency and even better understanding of the questions being asked. Approaches to combine these perspectives will be explored as a way to assess at-work disability that could be more comprehensive than using each of these approaches individually. Testing feasibility and performance of such a dual scaling work measure in persons with arthritis has been initiated.

Work-stream 3

Contextual factors have been defined by the OMERACT Filter 2.0 development team as “variables that are not the primary focus of the study but relate to the scope or setting of the study, and may include potential confounders and effect modifiers.” This has significant relevance to applications of worker productivity outcomes because a complex interaction between worker- and workplace-related factors (i.e., personal and environmental factors in the WHO International Classification of Functioning, Disability, and Health framework) is an important basis of health-related work functioning³². Just as worker well-being can influence work productivity, changes in job-related factors beyond the control of the worker (e.g., job demands) can also have significant influences on worker productivity. The need to consider potential contextual factors in clinical trials and cohort studies when interpreting worker productivity outcomes should be recognized. The main objectives of this work-stream are to (1) establish criteria to help define what would constitute a contextual factor of worker productivity, (2) achieve consensus on the best approach to measure these contextual factors, and (3) develop guidelines on how to consider and/or incorporate contextual factors in research analysis when worker productivity is a study outcome.

Some preliminary progress has been made at previous OMERACT meetings and from ongoing literature reviews³. To date, we have identified up to 70 potential contextual factors and have organized them into 2 major domains (personal or environmental factors), and also further into 1 of 15 subdomains: health, demographic attributes, economic need, personal appraisal, related skills/abilities, work-life balance, accessibility, economic climate/labor regulations, nature of work, workplace support, non-workplace support, organizational policies and practices, physical environment, work accommodations, and others³. The large number of potential contextual factors dictates the need to more closely examine whether some are more critical (i.e., core) to consider than others during various applications of worker productivity outcome measures.

Research Agenda for Work-stream 3

Several ideas and issues related to the classification and evaluation of contextual factors were discussed among conference participants during individual breakout sessions at the OMERACT 11 Workshop, which will guide our efforts toward this work-stream’s longterm objectives. These include (1) explorations of theoretical frameworks that could be applied to help organize the large breadth of potential contextual factors (e.g., International Classification of Functioning, Disability and Health), (2) approaches to adapt existing guidelines (e.g., Cochrane Collaboration, Grading of Recommendations Assessment, Development and Evaluation to help evaluate the quality of prognostic studies to grade level of evidence for specific contextual factors, and (3) addressing the need for usable guidelines to determine the relevance of specific contextual factors for different applications of worker productivity outcomes (e.g., RCT vs longitudinal observational studies).

Work-stream 4

The aim of this work-stream is to advance the interpretability of worker productivity outcome measures, which remains an area with limited understanding. A number of interpretability needs have been identified by our working group. For example, applications of worker productivity outcomes in clinical trials might benefit from a “response criterion” to facilitate responder analyses; interpretation of longitudinal observational studies might benefit from an improved understanding of what constitutes a “meaningful” between-group difference or within-group change beyond statistical comparisons of scores; and finally, some understanding of what constitutes a meaningful change at the individual level might also be useful for applications in clinical settings (e.g., vocational decision making). To these ends, we are keen to advance the interpretability of worker productivity measures, by building on efforts that have been initiated by others³³. Our aim is to draw insights from past OMERACT meetings to inform approaches to estimate cut scores associated with PAS^10,11 and magnitude of a score change that would represent the MID^12,13. To adapt to our concept of interest, we have introduced the new acronym of PAS-WP, which stands for patient acceptable state in worker productivity. An internal vote among working group members at a 2011 meeting in Toronto supported the pursuit of these objectives (Table 7).

Toward a Methodology for PAS/MID Estimations

Two central methodological issues related to PAS-WP and MID estimations are the choice of external anchor (to serve as “gold standard” reference comparator), and the specific cut-score estimation approach to be applied (e.g., 75% centile approach, use of receiver-operating characteristic curves). Prior to OMERACT 11, our working group engaged in a 4-round Delphi group exercise to identify specific PAS-WP and MID anchors to be used as reference comparators in our CAN-funded (discrimination) project. Candidate anchors were initially gathered in 2 ways: adaptation of existing anchors from the literature into anchors that describe a state or change in worker productivity, or generation of new PAS-WP and MID external anchors by Delphi participants. PAS-WP anchors derived from our Delphi exercise were discussed at selected individual breakout sessions. After providing a brief overview of the conceptual bases and methodological issues related to PAS-WP, we asked conference participants to assess the face validity of each of 7 anchors from our shortlist (Table 8). Discussions revealed that in general, participants favored PAS-WP anchors that made a clear distinction between acceptable and unacceptable states (e.g., PAS 5, where only 2 response options are available) and used simpler phrasing (e.g., PAS 7). Last, we presented preliminary PAS-WP estimates for various candidate presenteeism measures from our ongoing CAN-funded project, and highlighted the variability in cut-scores based on the different anchors applied.

Research Agenda for Work-stream 4

Feedback gathered from conference participants at OMERACT 11 will be used to refine our methodology (preferred external anchor, estimation method) to be applied when estimating PAS-WP and MID cutpoints for worker productivity measures as an objective of the CAN-funded study. These results will help contribute to filling the gap in evidence for the discrimination criterion of the OMERACT filter. In addition, our group will explore whether cut-scores associated with an “acceptable” productivity state can be further informed by seeking perspectives from others, such as employers or co-workers.

Feedback gathered from conference participants will inform the future activities of the working group. The following summarizes progress at OMERACT 11 and the working groups agenda.

Progress at the OMERACT 11 Workshop

• Review of the content and current evidence toward the OMERACT filter (truth, discrimination, and feasibility) of candidate global and multi-item measures of at-work limitations/productivity (i.e., presenteeism)

• Plenary vote to determine whether candidate global measures of at-work limitations/productivity currently meet the requirements of the OMERACT filter

• Endorsement of both presenteeism global measures from the Work Productivity and Activity Impairment questionnaire (item 5) and the RA-specific Work Productivity Survey (item 4) as having met the requirements of the OMERACT filter (received ≥ 70% support)

• Consideration for OMERACT endorsement of the presenteeism global measure from the Work Ability Index (item 1) as a candidate measure of work ability/difficulty at work (received ≥70% support)

Research Agenda of the Worker Productivity Working Group

• Ongoing literature reviews to update OMERACT filter evidence for candidate measures of worker productivity, with particular focus on the discrimination criterion where some gaps remain (e.g., emerging RCT)

• Completion of ongoing EULAR- and CAN-funded projects (led by working group members) to further explore patient appraisal, psychometric properties, and score interpretability of worker productivity outcome measures through cognitive testing, qualitative studies, and cohort studies

• Work with OMERACT to refine approaches to assess the merit of outcome measures that weigh both the quantity and methodological quality of available measurement evidence

• Development of criteria to identify and classify key contextual factors of worker productivity and guidelines to apply them for various clinical/research applications

• Exploration of the need for additional work-streams focused on issues related to the economic evaluation (monetization) of productivity loss