Decision-making: choose the right tool for the job

More often than not, decisions today are made by teams, based on a broad number of collected ideas and opinions. Collecting these inputs and then reaching a consensus can be made through a broad number of decision-making techniques. In this article we analyze five of the most popular ones: Prediction Markets, Surveys/Polls, the Delphi Method, the Nominal Group Technique, and the venerable Face-to-face Meeting.

Each one of these tools has its own merits and limitations, and there is no one-size-fits-all solution. The decision tree presented in Exhibit 1 will help you choose the best tool for the job at hands. Generally speaking:

• Prediction markets are particularly useful for collecting input from a large number of people, especially when it is important to rapidly incorporate new information. However, this technique is only effective with “Yes or No” questions;
• For multi-response or open-ended questions, a Survey or Poll might be a more viable option, provided that the question can be made clearly and factually: evidence shows that even slight changes in wording can lead to vastly different results;
• The Delphi Method is usually used to address complex issues, or matters that require quantified answers. The structured discussion process used in this method has proven to be effective in bridging gaps in assumptions and in mitigating bias. The Delphi can however be time-consuming and becomes cumbersome with more than 10 respondents;
• The Nominal Group Technique is often used at workshops to generate and prioritize ideas (e.g. In which products should we focus our R&D efforts on?). Compared with the Delphi Method, the Nominal Group Technique remains practical with more than 10 participants, but is less efficient in eliminating bias and in bridging gaps in assumptions;
• To tackle multiple issues at once, and do to so with minimal preparation time, the traditional Face-to-face meeting continues to be a valid option. Often described as an unstructured discussion (as opposed to the more rigid structure of the methods above), a decision-making meeting should nevertheless follow the same rules as any other effective meeting.

Let us now look into each one of these methods in more detail:

Prediction Markets

A prediction market turns events (e.g. “Will candidate A win the elections?” or “Will cold fusion be viable before 2020?”) into tradable securities: just like in a stock market, participants buy securities if they think the event will occur, and sell them if they think the event will not occur (with real money in some markets, and play money in others). The security price is then converted into an implied probability. For instance, a 0,8€ price for a security that rewards 1€ at maturity implies a probability of 80% that the event will indeed occur.

Research has questioned the validity of directly converting prices into probabilities, but prediction markets have nonetheless proven to be effective in collecting the “wisdom of the crowds” in order to predict the outcome of events. For instance, Exhibit 2 compares prediction markets and polls for projecting the outcome of US presidential elections, suggesting that the former is more accurate, especially when forecasting with more than 100 days in advance.

The use of prediction markets in a business context is still not widespread but has gained some traction over the last years. Inkling, for instance, is used by companies such as Procter & Gamble and Ford to bring together employees to evaluate ideas and assess risks. The biggest obstacle for the corporate use of prediction markets is arguably engaging the appropriate number of participants: the process itself requires a sufficiently large number of regular participants to ensure liquidity, while some company issues might be too specific or too sensitive for widespread discussion. 

Surveys and Polls

Surveys and Polls might not have the accuracy of Prediction Markets, but they hold important advantages: they work with a wide range and number of questions, they require minimal setup time, and most respondents are already familiarized with the process.

When preparing a survey or a poll, structuring the questions in a concise, clear and factual form is critical, as even small variations in wording can lead to vastly different results (Exhibit 3). It is also important to ensure that the responses are statistically valid i.e., that the number of responses is enough to ensure a reasonable sampling error, and that the characteristics of the respondents are representative of the target population.

The Delphi Method

The RAND Corporation originally developed the Delphi Method in the 1950s to forecast the impact of technology in warfare. Today this method is widely used to address complex issues, particularly those that require quantified answers. At X&Y Partners we routinely use it for market forecasting, for instance.

In a Delphi, a group of experts is asked to individually answer a question (e.g. “What will be the cost of solar energy in 2015?”). The answers and assumptions of the experts are then discussed and compared, and each of the experts makes a second prediction. This second prediction is also made individually, but now the expert is able to leverage the new information gained from the other experts. The process is iterative and usually runs until the moderator is satisfied with the degree of convergence of the answers from the various experts. One of the key characteristics of the Delphi is that the answers are anonymous: experts can see and discuss the responses from other experts, but do not know who is responsible for a particular answer. This contributes to eliminate peer pressure and other bias that might arise when some of the respondents are more influential than others.

Recent research has suggested that for quantitative judgment tasks the Delphi Method is slightly more accurate than Prediction Markets and the Nominal Group Technique (Exhibit 4), but less user friendly than the latter (Exhibit 6).

The Nominal Group Technique

Most of us have at some point participated in a workshop based on the Nominal Group Technique. In its basic form, participants are asked (individually or in smaller groups) to answer a question or to provide solutions for a particular issue (e.g. “How can we increase sales of our X product?”). The facilitator will then list all answers and foster a discussion to merge similar entries, clarify wording, and ensure that everybody has a similar understanding of the resulting propositions. Participants are then asked to rank these propositions, in order to identify the ones with a wider base of consensus. Exhibit 5 illustrates the results of a Nominal Group Technique, where participants were first asked to identify company stakeholders, and then to rank each stakeholder in terms of company influence and dependency.

For quantitative judgment tasks, the Nominal Group Technique is considered to be slightly more accurate than Prediction Markets but less so than the Delphi Method (Exhibit 4). In terms of ease of use, it is only outranked by the traditional Face-to-Face meeting (Exhibit 6).

Preparing a Nominal Group Technique can be time consuming, as this method requires a carefully planned structure and adequate facilities. It is also important to note that, even with an experienced facilitator, it can be hard to guarantee a full participation from the quieter contributors. Knowing the Meyers-Briggs Type (Exhibit 7) of each participant is often a valuable tool in preparing a Nominal Group Technique.

Face-to-Face meeting

All of the methods discussed above are fairly rigid: they require preparation time and follow a structured path that, for better or for worse, allows little room for deviation. 

If you have answered “No” to all questions in the decision tree from Exhibit 1, the unstructured discussion of a traditional Face-to-Face meeting might be the most suitable option to address the issue at hands. A lack of formal structure does not mean however a lack of organization. A decision-making meeting should follow the same guidelines of any other effective meeting: plan ahead, select the most appropriate participants, agree on an agenda, keep within the allotted time, wrap up the conclusions and follow-up actions, and distribute meeting minutes.