Bayesian Priors
Methods to elicit beliefs for Bayesian priors: a systematic review

https://doi.org/10.1016/j.jclinepi.2009.06.003Get rights and content

Abstract

Objective

Bayesian analysis can incorporate clinicians' beliefs about treatment effectiveness into models that estimate treatment effects. Many elicitation methods are available, but it is unclear if any confer advantages based on principles of measurement science. We review belief-elicitation methods for Bayesian analysis and determine if any of them had an incremental value over the others based on its validity, reliability, and responsiveness.

Study Design and Setting

A systematic review was performed. MEDLINE, EMBASE, CINAHL, Health and Psychosocial Instruments, Current Index to Statistics, MathSciNet, and Zentralblatt Math were searched using the terms (prior OR prior probability distribution) AND (beliefs OR elicitation) AND (Bayes OR Bayesian). Studies were evaluated on: design, question stem, response options, analysis, consideration of validity, reliability, and responsiveness.

Results

We identified 33 studies describing methods for elicitation in a Bayesian context. Elicitation occurred in cross-sectional studies (n = 30, 89%), to derive point estimates with individual-level variation (n = 19; 58%). Although 64% (n = 21) considered validity, 24% (n = 8) reliability, 12% (n = 4) responsiveness of the elicitation methods, only 12% (n = 4) formally tested validity, 6% (n = 2) tested reliability, and none tested responsiveness.

Conclusions

We have summarized methods of belief elicitation for Bayesian priors. The validity, reliability, and responsiveness of elicitation methods have been infrequently evaluated. Until comparative studies are performed, strategies to reduce the effects of bias on the elicitation should be used.

Section snippets

Background

What is new?

What this adds to what was known?

  1. This article summarizes methods that have been applied for belief elicitation;

  2. Reviews the published measurement properties of each method;

  3. Presents a conceptual framework for the belief-elicitation process;

  4. Identifies pragmatic methodologic strategies to reduce the effect of bias in belief-elicitation studies.

What should change now?
  1. Strategies to reduce the effect of bias include sampling from groups of experts, use of clear instructions and a standardized script, provision of examples and training

Search strategy

Eligible studies were identified using MEDLINE (1950 to week 2, June 2008), EMBASE (1980 to week 25, 2008), CINAHL (1982 to week 2, June 2008), Health and Psychosocial Instruments (1985 to March 2008), Current Index to Statistics (1974 to June 2008), MathSciNet (1940 to June 2008), and Zentralblatt Math (1868 to June 2008) using the search terms (prior OR prior probability distribution) AND (beliefs OR elicitation) AND (Bayes OR Bayesian). Mapping of term to subject heading was used, where

Search strategy

Systematic review of the literature identified 33 articles which described unique methods for belief elicitation in a Bayesian context (Fig. 1).

Study characteristics

Table 1 summarizes the study characteristics. Belief elicitation mostly occurred in cross-sectional studies (91%), at the level of the individual (97%), using small sample sizes (median of 11 participants). Questionnaires were largely administered in person (58%) or on paper (52%), and to derive a point estimate with individual-level variation (58%).

Elicitation methods

Discussion

This systematic review summarizes methods of belief elicitation for use in a Bayesian analysis. The validity, reliability, and responsiveness of the methods have not been adequately evaluated. Identification of the “best” method based on the principles of measurement science is limited by the paucity of data. With the increasing use of Bayesian analysis in clinical research [1], evaluation of the measurement properties of elicitation methods is required in order for researchers to be confident

Conclusion

This systematic review of the literature summarizes methods of belief elicitation for a Bayesian analysis. The measurement properties of the methods have not been adequately evaluated. Further evaluation of the validity, reliability, and responsiveness of elicitation methods is needed. Until comparative studies are performed, methodologic strategies to reduce the effect of bias on the validity and reliability of the elicited belief should be used. Based on the results of this systematic review,

Acknowledgments

Dr. Sindhu Johnson has been awarded a Canadian Institutes of Health Research Phase 1 Clinician Scientist Award. Dr. Brian Feldman is supported by a Canada Research Chair in Childhood Arthritis.

References (70)

  • D.A. Berry

    Bayesian clinical trials

    Nat Rev Drug Discov

    (2006)
  • D.J. Spiegelhalter et al.

    An overview of the Bayesian approach

    (2004)
  • K. Chaloner et al.

    Quantifying and documenting prior beliefs in clinical trials

    Stat Med

    (2001)
  • B.P. Carlin et al.

    Bayesian approaches for monitoring clinical trials with an application to toxoplasmic encephalitis prophylaxis

    Statistician

    (1993)
  • I.R. White et al.

    Eliciting and using expert opinions about influence of patient characteristics on treatment effects: a Bayesian analysis of the CHARM trials

    Stat Med

    (2005)
  • L.A. Moye

    Bayesians in clinical trials: asleep at the switch

    Stat Med

    (2008)
  • D.J. Spiegelhalter

    Incorporating Bayesian ideas into health-care evaluation

    Stat Sci

    (2004)
  • J.A. Singh et al.

    Development of classification and response criteria for rheumatic diseases

    Arthritis Rheum

    (2006)
  • S.R. Johnson et al.

    The health assessment questionnaire disability index and scleroderma health assessment questionnaire in scleroderma trials: an evaluation of their measurement properties

    Arthritis Rheum

    (2005)
  • D.L. Streiner et al.

    Health measurement scales

    (2003)
  • M.H. Liang

    Longitudinal construct validity: establishment of clinical meaning in patient evaluative instruments

    Med Care

    (2000)
  • A.R. Feinstein

    The theory and evaluation of sensibility

  • R.D. Errington et al.

    High energy neutron treatment for pelvic cancers: study stopped because of increased mortality

    BMJ

    (1991)
  • G.R. Bergus et al.

    Clinical reasoning about new symptoms despite preexisting disease: sources of error and order effects

    Fam Med

    (1995)
  • K. Chaloner et al.

    Graphical elicitation of a prior distribution for a clinical trial

    Statistician

    (1993)
  • L.S. Freedman et al.

    The assessment of subjective opinion and its use in relation to stopping rules for clinical trials

    Statistician

    (1983)
  • K. Chaloner

    Elicitation of prior distributions

  • H.C. de Vet et al.

    A randomized trial about the perceived informativeness of new empirical evidence. Does beta-carotene prevent (cervical) cancer?

    J Clin Epidemiol

    (1993)
  • J.S. Evans et al.

    Background beliefs in Bayesian inference

    Mem Cogn

    (2002)
  • D.J. Spiegelhalter et al.

    Applying Bayesian ideas in drug development and clinical trials

    Stat Med

    (1993)
  • N. Flournoy

    A clinical experiment in bone marrow transplantation: estimating a percentage point of a quantal response curve

  • P.H. Garthwaite et al.

    An elicitation method for multiple linear regression models

    J Behav Decis Making

    (1991)
  • P.H. Garthwaite et al.

    Elicitation of prior distributions for variable selection problems in regression

    Ann Stat

    (1992)
  • D.H. Gustafson et al.

    Developing and testing a model to predict outcomes of organizational change

    Health Serv Res

    (2003)
  • M.D. Hughes

    Practical reporting of Bayesian analyses of clinical trials

    Drug Inf J

    (1991)
  • D.J. Spiegelhalter et al.

    A predictive approach to selecting the size of a clinical trial, based on subjective clinical opinion

    Stat Med

    (1986)
  • J.L. Hutton et al.

    Bayesian sample size calculation and prior beliefs about child sexual abuse

    Statistician

    (1993)
  • N.P. Johnson et al.

    Survey of Australasian clinicians' prior beliefs concerning lipiodol flushing as a treatment for infertility: a Bayesian study

    Aust NZ J Obstet Gyn

    (2006)
  • P. Jones et al.

    Changing belief in obstetrics: impact of two multicentre randomised controlled trials

    Lancet

    (1998)
  • J.B. Kadane

    Progress toward a more ethical method for clinical trials

    J Med Philos

    (1986)
  • J.B. Kadane

    An application of robust Bayesian analysis to a medical experiment

    J Stat Plan Infer

    (1994)
  • J.B. Kadane et al.

    Experiences in elicitation

    Statistician

    (1998)
  • H.P. Lehmann et al.

    Bayesian communication: a clinically significant paradigm for electronic publication

    J Am Med Technol

    (2000)
  • Y. Li et al.

    Experimental tests of subjective Bayesian methods

    Psychol Rec

    (2005)
  • R. Lilford

    Formal measurement of clinical uncertainty: prelude to a trial in perinatal medicine. The Fetal Compromise Group

    BMJ

    (1994)
  • View full text