Considerations when selecting a software package to administer the Delphi.
| Does the Software Package Provide the Following? | Considerations |
|---|---|
| Initial recruitment e-mail to participants to solicit willingness and consent to participate in all rounds of the Delphi | Response rates may be improved by sending an initial e-mail to potential participants outlining the purpose of the Delphi and the number of rounds planned. Example of text: “Thank you for agreeing to participate. It is very important that you complete the survey in each round, as the validity of the study could be compromised if participants drop out. If participants drop out because they feel their opinions are in the minority, the final results will overestimate how much agreement there is on the topic6.” |
| Reminders (e.g., e-mail, other) | Only send to those who have not completed a round. Can these e-mails be personalized? A recent study suggests repeated reminders by e-mail, phone, and texts are acceptable to participants and produced high response rates18. Important to get all contact information at the recruitment stage. |
| Ability for participants to add domains in the first round | In many studies, participants should have an opportunity to add new domains in the first round. |
| Ability for the administrator to modify the list of domains between rounds based on results of the previous round | Does the software send all the domains for re-scoring or is there the option to provide a summary with a list of domains that achieved consensus (i.e., important based on a priori definition), domains that were removed (i.e., not important) and the list of domains to be re-scored? Is there an opportunity to clarify which domains were combined, reworded, or added? Summary lists, if provided, would be given in the final round, because it is important that participants have an opportunity to re-score domains after consideration of the feedback of others. |
| Feedback to individual participants | For each domain, how are individual participant scores and the distribution of scores from other participants displayed? Consider whether scores will be provided as aggregate or broken down by participant group (e.g., patients and other participants). Can participants include qualitative data (i.e., written comments)? |
| Ability to apply consensus a priori and allow for ranking | How much flexibility is there in defining consensus and can it vary between rounds? For example, in the initial rounds the participants would be asked to score domains, but in the final round they would be asked to rank domains in order of importance. Example for scoring: rating scale 1–9; if 70% of all participants select 7–9 (very important), the domains will be kept. If 70% of participants select 1–3 (not important), the domains will be dropped. Example for ranking: if you have a large number of domains that have achieved consensus, and want to reduce them to a manageable number, have participants select and rank their “top 10” in the final round. |
| Ability to extract pertinent data from the system | Is all the data entered by participants easily downloadable in a useful format? Is the data anonymized to maintain the spirit of the Delphi process? |