A valid and reliable belief elicitation method for Bayesian priors

Objective: Bayesian inference has the advantage of formally incorporating prior beliefs about the effect of an intervention into analyses of treatment effect through the use of prior probability distributions or "priors." Multiple methods to elicit beliefs from experts for inclusion in a Bayesian study have been used; however, the measurement properties of these methods have been infrequently evaluated. The objectives of this study were to evaluate the feasibility, validity, and reliability of a belief elicitation method for Bayesian priors.

Study design and setting: A single-center, cross-sectional study using a sample of academic specialists who treat pulmonary hypertension patients was conducted to test the feasibility, face and construct validity, and reliability of a belief elicitation method. Using this method, participants expressed the probability of 3-year survival with and without warfarin. Applying adhesive dots or "chips," each representing 5% probability, in "bins" on a line, participants expressed their uncertainty and weight of belief about the effect of warfarin on 3-year survival.

Results: Of the 12 participants, 11 (92%) reported that the belief elicitation method had face validity, 10 (83%) found the questions clear, and 11 (92%) found the response option easy to use. The median time to completion was 10 minutes (5-15 minutes). Internal validity testing found moderate agreement (weighted kappa=0.54-0.57). The intraclass correlation coefficient for test-retest reliability was 0.93.

Conclusion: This method of belief elicitation for Bayesian priors is feasible, valid, and reliable. It can be considered for application in Bayesian clinical studies.