The Delphi Technique

One of my first statistics consultations was for a graduate student using the Delphi Technique. I had never heard of it but was assured that the statistical part of it would not be difficult. Part of me was fascinated by learning a new technique; the other part was hoping that I might gain some prophetic insights as if making a pilgrimage to the ancient Oracle of Delphi in Greece.

What is the Delphi Technique? At first glance, it seems like a qualitative technique. You sit down with a group of experts to discuss something. That's a qualitative focus group method, right? Ah...but we take things a step further.

Suppose that I want to know how arts educators feel about standardized testing within their field. I might assume they would be against it since such testing in other areas is perceived to distract from their area. On the other hand, they might be for it since it would heighten validity for their field by being included alongside other subjects such as math and writing.

I start the technique by having experts reply to open-ended questions such as, "How do you feel about including music on your state's standardized annual assessment? Describe in detail." Now, that is definitely qualitative! But, then I take the responses and use them to develop a survey with response options that are quantitative (e.g., a Likert scale). Alternatively, I might identify shared themes and then construct the survey such that themes can be rank-ordered in terms of things like favorability and priority.

Next, I move on to what we would call Round 2. The same experts respond to the survey developed from the open-ended questionnaire. Now, this is still looking like some mixed-methods twist on a focus group technique. What sets this method apart from others is Round 3.

In Round 3, I actually share the results with the experts. In other words, Expert A gets to see the responses from Expert B, C, and D. They then take the survey again thereby having the opportunity to adjust their previous responses. This step is usually repeated once more (i.e., Round 4); however, the process could continue through several iterations.

Statistical analysis enters here when trying to determine an optimal level of agreement or merely wanting to display the results. I could use means and standard deviations for this. However, given the relatively small sample size used here, I might want to use the more sophisticated Score interval adapted by Randall Penfield at the University of Miami and myself for this process (Miller & Penfield, 2005).

The Delphi technique is an excellent technique for studies involving expert agreement and/or consensus building. It combines a traditional qualitative approach with the potential for rigorous quantitative analysis. Finally, it answers research questions that often have practical implications.

The Delphi Technique was created by the Rand Corporation.
For consultations, please send an e-mail to
Dr. Miller