It is the often untold story of project evaluations: We can never be fully certain about the results. Whether it is about the selection of the best alternative for a project or about setting priorities among different projects, the results depend to a considerable extent on assumptions we make. Assumptions are embedded in our analytical choices and results: what are the appropriate weights for each factor in a multicriteria analysis? How about the discount rate in a BCA? How accurate are the data sources we rely on? Do we truly know how much a project will cost or the level of future demand?
To be unaware of uncertainties in evaluation results means missing important pieces of information that can help support smart decision-making. Tackling this challenge, Mark Sieber, Chandler Duncan and Naomi Stein of EDR Group presented a poster at the TRB Annual Meeting in D.C., whose purpose was to show a number of different ways to deal with uncertainty in evaluations.
The poster demonstrated how when presenting the results of evaluations, it helps to avoid the appearance of deterministic accuracy and to account for both the multiple sources of uncertainty, and their respective scopes. Scenario-analysis and exploring sensitivities can help make evident that results consist of ranges of values rather than of one single value.
The focus on uncertainty can be an uneasy message for decision makers. Prioritizations are easier when you can entirely rely on, and rank based on, a single result. At the same time, results that do consider uncertainty are of no use to decision-makers unless they are accompanied by practical decision-oriented interpretation that help differentiate those conclusions that are possible given the available information from those which should be avoided. The TRB poster presents a number of techniques available to support decision-making in this context. Going forward, EDR Group is determined to follow up on this often-underestimated topic.
The poster was the result of an exchange enabled through EDR Group’s affiliation with EBP in Zurich, Switzerland. EDR Group’s Naomi Stein and Chandler Duncan teamed up with Mark Sieber, who joined EDR Group from EBP in 2016, to learn from their respective project experiences in the U.S. and Switzerland about appropriate ways to deal with uncertainty.