Where is the theory? Evaluating the theoretical frameworks described in decision support technologies

https://doi.org/10.1016/j.pec.2007.12.004Get rights and content

Abstract

Objective

To identify and describe the extent to which theory or theoretical frameworks informed the development and evaluation of decision support technologies (DSTs).

Methods

The analysis was based on the decision technologies used in studies included in the Cochrane systematic review of patient decision aids for people facing health screening or treatment decisions. The assumption was made that DSTs evaluated by randomized controlled trials, and therefore included in the updated Cochrane review have been the most rigorously developed.

Results

Of the 50 DSTs evaluated only 17 (34%) were based on a theoretical framework. Amongst these, 11 decision-making theories were described but the extent to which theory informed the development, field-testing and evaluation of these interventions was highly variable between DSTs. The majority of the 17 DSTs that relied on a theory was not explicit about how theory had guided their design and evaluation. Many had superficial descriptions of the theory or theories involved. Furthermore, based on the analysis of those 17 DSTs, none had reported field-testing prior to evaluation.

Conclusion

The use of decision-making theory in DST development is rare and poorly described. The lack of theoretical underpinning to the design and development of DSTs most likely reflects the early development stage of the DST field.

Practice implications

The findings clearly indicate the need to give more attention to how the most important decision-making theories could be better used to guide the design of key decision support components and their modes of action.

Introduction

Although there is increasing interest in supporting the participation of patients in decision-making by the use of decision support technologies (DSTs), there is also a poor understanding of how DSTs achieve their impact on outcomes [1]. While the number of published DSTs has tripled since 1999 [2], there is a growing concern that development has been independent of relevant theoretical frameworks. Over the past decade, little attention has been given to the theoretical foundation underlying the development of heterogeneously developed DSTs [2]. Although there are guidelines and criteria being produced for the design and evaluations of DSTs [3], their development process, content and evaluation does not seem to recognise the need to adhere to any conceptual or theoretical framework relevant to decision-making.

The terms “theory” and “model”, often confounded, are associated with an overwhelming variety of definitions [4]. However, these are independent concepts that need to be carefully defined and distinguished. In the present study, we chose to define theory as a set of inter-related propositions (theoretical constructs) that constitute a framework for describing, explaining and predicting the decision-making process. Theories propose to explicate the rules and mechanisms by which the outcomes are achieved. Compared to a model, a theory tends to address global behaviours in general context and to be discipline specific. According to Hawking [5], a good theory “satisfies two requirements: it must accurately describe a large class of observations on the basis of a model that contains only a few arbitrary elements, and it must make definite predictions about the results of future observations”. Models are informed by one or more theories and have a very limited capacity to predict behaviours. A model may also include processes or constructs that are not based on theory. Finally, models make an extensive use of representations in describing a phenomenon or the interactions between a set of constructs. While theories describe and explain behaviour in an attempt to predict it, models are essentially descriptive.

Most interventions in this field appear to have been developed in a practical manner, using a wide range of medias, timeframes and purposes. DSTs have also been noted to achieve different levels of effectiveness [6]. More importantly, the design of the majority of DSTs seems to be primarily informed by researchers who create products that combine information and graphical elements to portray risk but generally lack a theoretical hypothesis about how patients will achieve decisions, with or without health professionals [7]. The potential impact of a theoretical foundation on the quality and efficacy of DSTs has never been formally assessed.

Health conditions are often associated with several treatment or screening options, each involving significant levels of harms and benefits. Decisions to undertake a treatment or a screening test depend on the differences between the harms and benefits of each option and how these are valued and evaluated by patients and their clinicians. As a consequence, the patient's perspective needs to be taken into account. Contexts such as these give rise to the need to involve patients in deciding on their care in order to make optimal decisions that are ideally consistent with their knowledge, values and long-term goals. To achieve these goals, there is increasing interest in developing technologies that support patients when they face tough decisions, for themselves or others in their families. Those interventions, referred to here as DSTs (also known as patient decision aids) provide information about the treatments or screening options made available to patients. They are designed to help patients choose between two or more courses of action by providing information about the probabilities associated with the risks and benefits of each option.

A literature review of health technologies intended to influence patient informed decision-making assessed the theoretical basis of 547 studies of interventions ranging from the comparison of information mediums or simple provision of additional information to the use of DSTs [8]. The research was not exclusively focussed on DSTs and did not address the question as to what extent theory had guided the development and evaluation of the DSTs. The findings showed that theory was not frequently used in health technologies. Indeed, 82% of the studies did not refer to, or make use of any theory. Amongst those which did explicitly refer to decision-making theories, there was little account of the way in which the theory had been used. For instance, there was no clear specification of how the theoretical concepts or framework described were subsequently applied to the practical design of the DST.

Another study investigated the theoretical basis of interventions designed to promote patients’ informed decision-making in the clinical context of cancer screening [9]. The findings showed that 5 of a total of 14 interventions referred to a theoretical framework. However, among those interventions that were described as “theory-based”, it was not clear how the specified theory had shaped the design of the specific intervention. None of the articles reporting the development and evaluation of the DSTs commented on the utility of the chosen theoretical foundation.

To conclude, there is no clear description of a deliberate avoidance of theory nor is there detailed attention to how some, albeit a minority, used a specific theory for design, development and evaluation. Similarly, the effect of a theoretical foundation on the impact and efficacy of a DST has not been formally assessed. DSTs that are not based on theory may be as efficient and reliable as interventions guided by a relevant theoretical basis, however, for the time being we are unable to assess this area. The aim of the present study is to describe and analyse rigorously developed DSTs in order to determine the contribution of decision-making theory to their conception, design, development and evaluation. As a sample frame, we reviewed the 55 published randomized controlled trials of DSTs included in the Cochrane systematic review [2].

Section snippets

Inclusion criteria

Our sample frame was the 55 trials of “patient decision aids for people facing health treatment or screening decisions”, included in the Cochrane systematic review. The assumption was made that DSTs evaluated by randomized controlled trials included in a Cochrane review would have been among those most rigorously developed. In the Cochrane review 22,778 citations were identified and 55 randomized controlled trials of DSTs were extracted and included in the review. The interventions focussed on

Results

In the present study, 78 full text articles reporting the development and evaluation of the DSTs in a randomized controlled trial were reviewed. In total, 55 trials of “patient decision aids for people facing health treatment or screening decisions” were included in the Cochrane systematic review. However, the authors note that three DSTs [11], [12], [13] have been evaluated in two or more trials [14], [15], [16], [17], [18]. There may have been small changes between versions but we made the

Discussion

The analysis of 50 DSTs that have been subjected to evaluation by randomized controlled trial showed that only a third had described the contribution of decision-making theories or models to their design, development and evaluation. When these 17 DSTs were considered in depth, there was little evidence that developers were basing the design, construction and evaluation of DSTs in line with theoretical principles and predictions. The analysis of all publications reporting the development and

Acknowledgement

Funding: The Sir Halley Stewart Trust, Willingham, Cambridge, CB24 5LS, United Kingdom (PhD studentship).

References (117)

  • A.M. O’Connor et al.

    Do patient decision aids meet effectiveness criteria of the international patient decision aid standards collaboration? A systematic review and meta-analysis

    Med Decis Making

    (2007)
  • G. Elwyn et al.

    Developing a quality criteria framework for patient decision aids: an online international Delphi consensus process

    Br Med J

    (2006)
  • J.A. Earp et al.

    Conceptual models for health education research and practice

    Health Educ Res

    (1991)
  • S. Hawking

    The illustrated a brief history of time

    (1996)
  • V. Entwistle et al.

    Evaluating interventions to promote patient involvement in decision making: by what criteria should effectiveness be judged?

    J Health Serv Res Policy

    (1998)
  • B.K. Rimer et al.

    Informed decision making: what is its role in cancer screening?

    Cancer

    (2004)
  • H. Bekker et al.

    Informed decision making: an annoted bibliography and systematic review

    Health Technol Assess

    (1999)
  • D.J. Bowen et al.

    Theoretical foundations for interventions designed to promote informed decision making for cancer screening

    Ann Behav Med

    (2006)
  • R. Evans et al.

    Toward a model for field-testing patient decision-support technologies: a qualitative field-testing study

    J Med Internet Res

    (2007)
  • M. Gatellari et al.

    Does evidence-based medicine about screening for prostate cancer enhance consumer decision making?

    J Med Screen

    (2003)
  • M.J. Green et al.

    Effect of a computer-based decision aid on knowledge, perceptions, and intentions about genetic testing for breast cancer susceptibility: a randomized controlled trial

    J Am Med Assoc

    (2004)
  • S. Dodin et al.

    Prise de decision en matière d’hormonothérapie de remplacement

    Can Fam Physician

    (2001)
  • M. Gatellari et al.

    A community-based randomised controlled trial of three different educational resources for men about prostate cancer

    Patient Educ Couns

    (2005)
  • A.M. O’Connor et al.

    The effect of an “explicit” values clarification exercise in a woman's decision aid regarding postmenopausal hormone therapy

    Health Expect

    (1999)
  • M. Holmes-Rovner et al.

    Patient decision support intervention: increased consistency with decision analytic model

    Med Care

    (1999)
  • A.A. Montgomery et al.

    A factorial randomised controlled trial of decision analysis and an information video plus leaflet for newly diagnosed hypertensive patients

    Br J Gen Pract

    (2003)
  • M.L. Rothert et al.

    An educational intervention: increased consistency with decision analytic models

    Res Nurs Health

    (1997)
  • M.S. Van Roosmalen et al.

    Randomised trial of a decision aid and its timing for women being tested for a BRCA 1/2 mutation

    Br J Cancer

    (2004)
  • D. Bernoulli

    Exposition of a new theory on the measurement of risk (original: 1738)

    Econometrica

    (1954)
  • J.W. Pratt et al.

    The foundations of decision under uncertainty: an elementary exposition

    J Am Stat Assoc

    (1964)
  • J. von Neumann et al.

    Theory of games and economic behavior

    (1944)
  • D. Kahneman et al.

    Prospect theory: an analysis of decision under risk

    Econometrica

    (1979)
  • L.J. Savage

    The foundations of statistics

    (1954)
  • R.A. Howard et al.

    Readings on the principles and applications of decision analysis

    (1984)
  • J.F. Magee

    Decision trees for decision making

    Harvard Bus Rev

    (1964)
  • S.P. Pauker et al.

    The amniocentesis decision: an explicit guide for parents

    Birth Defects

    (1979)
  • H.L. Bekker et al.

    Applying decision analysis to facilitate informed decision making about prenatal diagnosis for down syndrome: a randomised controlled trial

    Prenatal Diag

    (2004)
  • J. Protheroe et al.

    The impact of patients’ preferences on the treatment of atrial fibrillation: observational studies of patient based decision analysis

    Br Med J

    (2000)
  • A. Robinson et al.

    The potential use of decision analysis to support shared decision making in the face of uncertainty

    Qual Health Care

    (2000)
  • J.G. Dolan et al.

    Randomized controlled trial of a patient decision aid for colorectal cancer screening

    Med Decis Making

    (2002)
  • M. Zeleny

    Multiple criteria decision making

    (1982)
  • C.L. Hwang et al.

    Multiple attribute decision making: methods and applications

    (1981)
  • J.S. Dyer et al.

    Multiple criteria decision making, multiple attribute utility theory: the next ten years

    Manage Sci

    (1992)
  • T.L. Saaty

    Multicriteria decision making: the analytic hierarchy process

    (1990)
  • A.G. Hunter et al.

    A randomized trial comparing alternatives approaches to prenatal diagnosis counseling in advanced maternal age patients

    Clin Genet

    (2005)
  • L. Lalonde et al.

    Evaluation of a decision aid and a personal risk profile in community pharmacy for patients considering options to improve cardiovascular health: the options pilot study

    Int J Pharm Pract

    (2006)
  • A. Shorten et al.

    Making choices for childbirth: a randomized controlled trial of a decision aid for informed birth after cesarean

    Birth

    (2005)
  • M. Fishbein

    Belief, attitude, intention, and behavior: an introduction to theory and research

    (1975)
  • A. Tversky et al.

    Advances in prospect theory: cumulative representation of uncertainty

    J Risk Uncertainty

    (1992)
  • I.L. Janis et al.

    Decision-making: a psychological analysis of conflict, choice and commitment

    (1977)
  • Cited by (89)

    View all citing articles on Scopus
    View full text