Abstract
Objective. At a previous Outcome Measures in Rheumatology (OMERACT) meeting, participants reflected on the underlying methods of patient-reported outcome (PRO) instrument development. The participants requested proposals for more explicit instrument development protocols that would contribute to an enhanced version of the “Truth” statement in the OMERACT Filter, a widely used guide for outcome validation. In the present OMERACT session, we explored to what extent these new Filter 2.0 proposals were practicable, feasible, and already being applied.
Methods. Following overview presentations, discussion groups critically reviewed the extent to which case studies of current OMERACT Working Groups complied with or negated the proposed PRO development framework, whether these observations had a more general application, and what issues remained to be resolved.
Results. Several aspects of PRO development were recognized as particularly important, and the need to directly involve patients at every stage of an iterative PRO development program was endorsed. This included recognition that patients contribute as partners in the research and not merely as subjects. Correct communication of concepts with the words used in questionnaires was central to their performance as measuring instruments, and ensuring this understanding crossed cultural and linguistic boundaries was important in international studies or comparisons.
Conclusion. Participants recognized, endorsed, and were generally already putting into practice the principles of PRO development presented in the plenary session. Further work is needed on some existing instruments and on establishing widespread good practice for working in close collaboration with patients.
The development and use of patient-reported outcomes (PRO) was a major topic at the Outcome Measures in Rheumatology (OMERACT) 10 meeting, where the workshop Choosing or Developing Instruments was designed to help participants reflect on the underlying methods of instrument development1. Tradeoffs between using current imperfect measures and the long and complex process of developing new instruments were considered, together with the need for rigor in PRO instrument development. As part of an agenda for action it was recommended that researchers and patient partners work together to tackle these issues, and that OMERACT bring forward proposals for acceptable instrument development protocols. It was intended that these would contribute to an enhanced version of the “Truth” statement in the OMERACT Filter, a widely used guide for outcome validation2. In response to that request, the present session at OMERACT 11 was designed to examine the issues experienced during practical application of rigorous PRO development principles, as would be required explicitly in the expanded formulation of the OMERACT Filter (called Filter 2.0) being proposed.
Since the previous OMERACT meeting, the US Food and Drug Administration (FDA) finalized its guidance on PRO development3 and drafted guidance for industry (currently distributed for comment purposes only): Qualification Process for Drug Development Tools4. In 2008 the Critical-Path (C-Path) PRO Consortium was created by collaboration between the FDA and pharmaceutical companies5. There has also been a more general acceptance of the need for rigor in defining PRO, partly in response to the participation in and publication of the previous OMERACT discussions1.
The purpose of the present OMERACT session was to explore to what extent the Filter 2.0 proposal was already being applied, whether it was practicable, and whether there were aspects that researchers would have difficulty achieving.
Presentations
There were 3 initial brief introductory presentations. E. Nikaï described the origins and progress to date of the C-Path PRO Consortium and more specifically the work undertaken within the rheumatoid arthritis (RA) working group (WG). The C-Path Institute was formed in 2005 by the University of Arizona and the FDA with the aim of implementing the FDA’s Critical Path Initiative (a strategy for transforming the way FDA-regulated products are developed, evaluated, manufactured, and used). Within this institute several consortia are active and one of them is the PRO Consortium. Membership is available to medical product companies, and the PRO Consortium was tasked with a mission to establish and maintain a collaborative framework with appropriate participants for the development of qualified, publicly available PRO instruments for use in clinical trials where PRO endpoints are used to support product labeling claims. The RA WG within the C-Path PRO Consortium was set up in early 2011 and this working group recognized the benefits of tapping into the previous and current work of OMERACT, including the outcome of the present conference.
V. Strand reviewed the way OMERACT had developed its own approach to PRO development, setting out as an example the identification of fatigue as an important domain in assessing RA outcomes: this was illustrated by the poor performance of traditional instruments for measuring fatigue, which was addressed by the rigorous development of a new fatigue scale and its subsequent good performance in practice6,7,8.
J. Kirwan illustrated pitfalls in PRO development when input from those involved (particularly from relevant patient groups) is absent. Recent structured interviews with patients had shown that a well-regarded questionnaire measuring the effect of foot involvement in RA omitted 3 substantial areas identified as important (e.g., whether a health professional had ever shown an interest in the patient’s feet), but which could be encompassed with simple additional questions such as “Has a health professional ever examined your feet, in relation to your RA?” (O. Wilson, manuscript in preparation.) In a further example, small but important changes had been introduced to a developing questionnaire on fatigue when cognitive interviewing or asking the patient to “think aloud” while completing the questionnaire9 had been used to assess patient understanding10. The steps required in good PRO development were reviewed, drawing particularly on the FDA guidance3, and showing how the recommendations of this and several other recent publications in the field11,12 coincided with the main issues that had emerged in a less well-defined way at OMERACT 10. A strategy was proposed for OMERACT and an article by Frost12 was distributed describing an approach (summarized in Figure 1) that mirrors the approach proposed for the whole of core set development in Filter 2.0. A substantial proportion of the development pathway is concerned with Truth within the OMERACT Filter.
Steps in creating and assessing patient-reported outcome instruments (after Frost, et al12).
Case Studies and Discussion Groups
Discussion (“breakout”) groups were asked to consider these points further, aided by 5 case studies drawn from working groups across the spectrum of OMERACT activity (Table 1). Each breakout group was a random selection of participants and made up of about 12 clinicians or clinical researchers, 4 industry personnel, and 5 researchers. In addition there were at least 2 patients in every breakout group. Breakout groups were invited to consider whether the PRO strategy applied to what they had heard in the case study, and whether it was more widely applicable and feasible. In addition, specific questions related to the case study presentation were also addressed (Table 1). These questions were not subject to a detailed or specific report back, but were rather used to stimulate the discussion. The focus of feedback was on whether the proposals for Filter 2.0 addressed the issues involved in these areas, and whether the proposed Filter 2.0 concepts or wording needed adjustment. From the reports back to the plenary session and their subsequent amalgamation by discussion between reporters it was clear that most current OMERACT PRO areas of work have already complied with the basic principles, and several broad issues emerged.
Summary of 5 case studies and related discussion group questions. Numeric values in square brackets indicate references.
The need to directly involve patients at every stage of PRO development was endorsed. There were no obvious circumstances in which the validity of a PRO could be ensured if some of the process was omitted. The example of fatigue in RA showed that patients made an indispensable contribution as participants (e.g., in focus groups and surveys), as research partners (e.g., in identifying important outcomes and interpreting results), and in several roles that sit between the two (e.g., in cognitive interviewing or formulating questionnaire items). There was a clear recommendation that cognitive interviewing would be an important step forward in clarifying the meaning of “patient global” as an assessment tool (although there was also a feeling that when used in a group setting such as a clinical trial, the present instruments were relatively robust). Further focus group work was also recommended to clarify patient global assessment as an outcome measure.
How best to work with patient research partners, from both a technical viewpoint and an interpersonal viewpoint, was considered by several breakout groups. Drawing on several experiences of patient involvement with working group activities between OMERACT meetings, participants noted the important role of working group leadership in facilitating patient engagement in the research process. The participants affirmed that ensuring patients are adequately briefed is essential to enable their full contribution, that it is vital to maintain patient involvement throughout the process, and that the process is applicable across conditions and domains. Explicit discussion with and acceptance by researchers and clinicians of appropriate patient participation was felt to be necessary to maximize this aspect of the research process13. One discussion group reviewed in detail the experiences of an OMERACT patient partner and noted that while there are many positive aspects to serving in this role, there are also some challenges. For example, it might be difficult to hear about increased mortality and the spectrum of difficulties faced by patients, and patient partners might be upset after reading focus group transcripts or after meetings in which these types of issues are raised.
Issues were addressed related to the language and cultural translations required for PRO to be comparable in different countries. Full cross-cultural validation implied substantial effort and resource commitment, and participants commented that the process is cumbersome and is perhaps not practical in all circumstances. On the other hand, verifying the meaning of questionnaire items in other cultures is very important. It was suggested that the developers of a PRO should consider “translatability” from the earliest development phase of an instrument, for example by avoiding idiomatic expressions and if possible by involving a bilingual person in the initial phase. Contextual factors such as socioeconomic status and culture may be difficult to solve even after proper translation. More research must be done regarding the need for back-translation. It might also be necessary to revise/update translations as there are inter-generational changes.
It was clear overall that participants were largely in agreement with the elements of PRO development outlined in the plenary presentation (Figure 1) and were keen to ensure that this strategy, as endorsed by the FDA, is incorporated explicitly into the OMERACT Filter 2.0 procedure.
