basic questions about “cost/benefit analysis”

Transit projects, like all government projects, have costs and benefits, so one classic way to evaluate project proposals has been the "Cost/Benefit Analysis" (CBA).  Add up all the benefits, add up all the costs, divide benefits by costs and announce the Benefit/Cost Ratio (BCR).  If the it's below 1, which means the costs exceed the benefits, you kill the project.  The higher the it is, the better the project.

In the Financial Post (part of Canada's National Post), Peter Shawn Taylor argues that the Cost/Benefit Analysis is the best way to evaluate a project, and that bad projects are being advanced by a newfangled thing called Multiple Account Evaluation (MAE).  I encourage you to be bored by this dispute, but not before you understand it.  Neither of these methods is sufficient to end a passionate argument, and it's important to know why they never will be.

The problem with Cost/Benefit analysis is that it requires you to convert all the costs, and all the benefits, to the same currency.  That means you must know, with imperial confidence, the cost in dollars of such things as:

  • each minute of each customer's time
  • a particular ecosystem to be destroyed or preserved, which may involve various degrees of endangerment (of species, and of ecosystem types)
  • historic or cultural resources to be destroyed or relocated, or preserved.
  • the redevelopment potential of a particular area with or without the project, and the various benefits and costs arising from that potential.
  • impacts of the project on affordability, and thus on the future shift of disadvantaged persons from one area to another, with a range of social impacts.
  • benefits of electrification (quiet, no on-site pollution) on a neighborhood's quality of life, which impacts the previous point.
  • a particular aesthetic impact that makes the city distinctive in a new way, such as a stunning piece of architecture or a new relationship to a unique historic artefact or feature of landscape.

Several of these items require confident prediction of the future, and there is also a vast question of who pays these costs and reaps these benefits.  I am not criticizing the vast body of research that has gone into improving the conversion factors that turn all costs and benefits into a dollar value.  But unless we really agree on what endangered ecosystems or municipal self-esteem are worth, in dollars, it's reasonable to question whether Cost/Benefit analysis can deliver the last, decisive word on whether a project deserves public investment.

One Canadian solution is the Multiple Account Evaluation (MAE).  Different types of benefits and costs are calculated differently and not converted into a common currency.  As Taylor describes it:

Developed by the [British Columbia] government in 1993 and now in widespread use, MAE dispenses with a single spreadsheet of advantages and disadvantages and adopts instead numerous separate “accounts:” a financial account, social account, environmental account and so on. In this way, the actual monetary costs and benefits of a project become just one of many issues to be considered.

A similar idea is inherent in the term "triple bottom line," which refers to economic, environmental, and social impacts — positive and negative — of a proposal.  In either case, you end up with a series of parallel analyses that give different answers from the point of view of different kinds of cost and benefit. 

So then what do you do?  Taylor quotes emeritus Professor John Shortreed of the University of Waterloo:

“The problem with MAE is that each account is given equal weight,” he observes. “This suggests the billion-dollar cost of the project is no more or less important than any of the other accounts, however trivial.”

It's easy to weight each account equally, because that sounds fair, but of course equal weighting is just as arbitrary as any other weighting, because we're talking about things that are not directly comparable to each other, such as a social cost vs an environmental one.  If you could really compare those things, we'd be able to do cost/benefit analysis.

MAE and the "triple bottom line" are useful concepts because they reveal the arbitrariness of weighting.  Weighting implies decisions, for example, about the relative importance of social vs environmental impacts.  That means the weighting is a value judgment about what matters to us as humans, as a community, as a civilization.  Surely we should argue passionately, maybe even irrationally, about that! 

The real problem here is that in the interests of consensus, we tend to allow technical analysis to make important value judgments for us, which is to say, we want technical analysis to tell us who we are. 

When you hear the terms social benefit, environmental benefit, and economic benefit, which arouses the strongest positive feelings?  The answer is an important signal about your deeply-held values and world-view.  If a technical analyis is making that decision for you, are you sure it's the analysis you want to trust?

Cost/benefit analysis and Multiple Account Evaluation (or "triple bottom line") both conceal value judgments.  Cost/benefit analysis hides value judgments in the factors used to convert various costs and benefits into dollars.  MAE or "triple bottom line," by contrast, comes up with multiple ratios — social vs environmental, for example — and ends up having to weight them, which is where the judgment appears.  I prefer the MAE or "triple bottom line" only becuase it makes the arbitrariness of weighting more visible, and hence pushes the conversation about it closer to the public sphere. 

It's understandable that we don't want to compare a billion-dollar pricetag to unquantifiable but powerful benefits that people will weight differently.  But the whole idea of the triple bottom line is that (a) we have to make these comparisons and (b) there's no technical basis for an answer.  When I'm doing evaluation frameworks of any kind, I look for guidance on the weighting based on locally adopted goals, value statements, or public discussion.  If the goals haven't been articulated, I often suggest some public discussion about the weighting itself.  (Some evaluation processes do consult the public on weighting, but the question can sound too technical at that point so not enough people pay attention; I think more can be done to make these discussions more vivid and consequential.)

As usual, I don't have the answer, only a refinement of the question.  Should communities talk about how to weigh competing values that are in conflict?  Or should they let those decisions be made inside a technical process in the guise of analysis?  There's a very powerful argument for the latter: decisions get made.  It's a lot of work to educate a community enough that it can express its values and desires in a form that a project can implement, even before you impose the other value judgments attached to various funding sources.  I don't advocate either position entirely, but have argued the "discuss the values" position because it should be visible as a choice.  In many ways, this is the "democracy vs technocracy" debate that we're hearing right now in the context of the European Union.  It's a hard question, maybe too hard, but it's one of those great debates worth caring about.

16 Responses to basic questions about “cost/benefit analysis”

  1. EngineerScotty December 5, 2011 at 10:23 pm #

    The fly in the ointment, of course, is that there is widespread disagreement on the weights of these things among the communities that a (potential) project may serve.
    If you are someone who will be paying taxes to support a transit line, but who won’t be using it (such as someone living in a place not directly served by the line), your view of the weighting will likely differ from someone who will be directly impacted by the project.
    Attempts to force a particular costing model as technically correct are more often than not attempts to place the thumb on the scale to achieve a predetermined outcome.

  2. Joshua December 6, 2011 at 2:16 am #

    I think this post raises a number of excellent points. There are fundamental flaws with the cost-benefit analysis process, because it has to turn everything into a dollar value for it to count. This is easy to do with some things (a few minutes faster for a lot of people adds up quick) but really difficult for other things (quality of urban environment, severance, pollution etc). What generally happens is the “too hard” stuff gets ignored.
    A classic example of this is the post you did a few months back on the Parisian tramway that apparently had a negative BCR, even though clearly it had vastly improved the area it was located in. The cost-benefit analysis valued a few seconds of delay for motorists on a motorway above a massive improvement in urban quality.
    I tend to think that transport decisions will always be political decisions – and that’s fine, as long as we’re transparent about it and as long as there’s some accountability. We have a huge problem in New Zealand that central government holds most of the transport purse-strings, yet it’s in local government elections that people think about transport when voting. A cost-benefit analysis is a useful part of assessing whether to proceed with something, but absolutely shouldn’t be the be-all and end-all.
    Generally, when forming an opinion on a transport project I ask some qualitative questions:
    1) Could the benefits (or some of them) of this project be achieved through something else that would cost less?
    2) What will the land-use outcome of this project be and is that consistent with our land-use plans?
    3) What are the environmental impacts of the project and are they acceptable?
    4) Does the project promote or reduce (and to what extent) car dependency?
    5) Who benefits from the project, who doesn’t benefit, how significant are the different benefits/disbenefits and how do they align with our ‘big picture’ goals for the area?
    The answers to these questions typically tell us more than a single cost-benefit ratio number.

  3. Morten Skou Nicolaisen December 6, 2011 at 3:37 am #

    Excellent post. In addition to the points raised by Jarred, I would like to add three crucial arguments against heavy reliance on CBA:
    1 – Distributional effects are ignored. Very mobile groups are typical recipients of benefits, while social and environmental costs are usually carried by local residents that are not necessarily very mobile.
    2 – Discounting makes important environmental impacts disappear, as the most dramatic effects are often far in the future. Some of these are potentially irreversible or carry an enormous path dependency, but CBA methodology naively assumes that we can simply buy things like extinct species or depleted forests back. On top of this comes the pricing issues Jarret raises, where an example could be CO2 quotas hitting a ludicrous price of zero during 2007.
    3 – In the transport sector CBA results are practically impossible to verify even after project completion, due to the intangible indicators that one must assign a monetary value. Travel time savings are usually the biggest benefit factor, but if we consider the Marchetti constant there seem to be little empirical evidence of any travel time savings as a result of new infrastructure. People simply travel longer as a result of reduced journey times. This means that on top of dubious unit pricing we are also faced with dubious unit measures. The ‘objectivity’ of CBA results are rarely objective at all, and is just one of many socially constructed yardsticks by which to measure performance.

  4. Morten Skou Nicolaisen December 6, 2011 at 3:39 am #

    … and I hope Jarrett can forgive me for misspelling his name twice in my previous post 🙂

  5. Alan Howes December 6, 2011 at 5:12 am #

    In the UK CBA (or COBA) has been dead for quite a while, its successors being NATA (New Approach to Appraisal) in England and STAG (Scottish Transport Appraisal Guidelines) in Scotland. No doubt something similar in Wales and NI too. All follow the same principles as MAE, and the abve discussion is history for us. I foolishly assumed that most of the civilised world was on the same track – is the US at the back of the pack?
    I’d be interested to hear of the situation in other countries.
    (Actually, England is now onto son of NATA, and STAG has been revised too – mainly in the interests of simplicity. And Stakeholder (including community) involvement is integral to both the systems.)

  6. Jason Kramer December 6, 2011 at 5:33 am #

    I would dispute your characterization that CBA “hides” value judgements in the factors to convert costs and benefits into dollars.
    In fact, CBA explicitly uses dollars to place value judgments on various factors. Of course, those amounts are uncertain and subject to debate, but that doesn’t mean they’re hidden.

  7. BruceMcF December 6, 2011 at 12:08 pm #

    One of the judgements intrinsic to CBA is the judgement that different types of costs and different types of benefits all have the same priority.
    And that judgement is indeed hidden behind the CBA question of “what cost shall we assign to this minute and what benefit do we assign to that new opportunity available to property developers?” … unless, of course, it is explicitly argued through and decided before performing the CBA what specific set of costs and benefits ARE of equuivalent priority, in which case the CBA is only one of several accounts in a multiple-account analysis.

  8. Jarrett at December 6, 2011 at 2:38 pm #

    Jason.  I mean "hidden" in the sense that their nature as subjective value judgments is hidden by the ostensibly quantified, single-score conclusion. 

  9. Murray December 6, 2011 at 3:37 pm #

    I’d be keen to read more about the MAE sometime. I am always reminded of something I read in a debate about Route 20 in Brisbane in the 1980s:
    You can compensate an old woman for the loss of part of her front yard, but how do you compensate for the fact that she no longer feels safe or confident crossing the road to visit her friends?

  10. Steve Munro December 7, 2011 at 11:07 am #

    One central flaw in the MAEs used within Metrolinx studies here in Toronto is the “benefit” assigned to the cost of building a line. The more expensive the option, the more spin-off effects you create in secondary and tertiary spending on labour and materials. Building a $1-billion line generates more “benefit” than a $500-million one, even though that money could have been used for a completely unrelated project. The benefit of building two or more smaller projects rather than one big one is never examined because each proposal is evaluated in isolation. The final decision comes down to the politics of which project has the most supporters, and the expensive ones tend to win.

  11. Zoltán December 7, 2011 at 12:41 pm #

    There are three particular problems with the workings of CBA:
    – It assumes a property right to the status quo, and thus tends to reinforce it. For example, if you’re someone that benefits from using a highway that’s subsidised by the government at an average cost of $5 per user, and a $5 toll is proposed, a CBA would assume a disbenefit to society of $5 per user, rather than questioning whether you really ought to have been paying $5 all along.
    – It usually establishes its values in an abstract way, most notably values of time. These are normally based upon stated preference surveys establishing how much people would be willing to pay for certain scenarios that cost or save them time. Having derived a value of time, it can multiply savings of say, 30 seconds over thousands of people, and come up with huge benefits. But no one has asked “How much would you be willing to pay to get to work 30 seconds quicker?”. If one did ask, the answer would probably be nothing.
    – Those abstract values are established by people like me, when I worked one summer some years ago doing surveys for an EU-funded study to establish willingness to pay for improved quality of water in rivers and lakes. People often didn’t understand the questions, and I couldn’t be bothered to explain, given the soul-crushing work and hours. Sometimes I filled things in for them so I could finish and go home. And often they double-checked that the cost scenarios I’d just put to them were hypothetical, and none of this would actually take place – because they wouldn’t want to pay more for water in real life. …defeating the object entirely.

  12. Rob December 7, 2011 at 8:53 pm #

    As usual I’m late to this discussion – so I may get the last word but nobody else will see it!
    This is a good post, thanks. I’d propose a more expansive principle still: value decisions should be made by people empowered to make them, not by technicians creating models. If you believe that, then the purpose of technical measures and evaluations is to inform the decision maker – and the black box that is benefit cost analysis fails to do that most of the time. Technical measures should be thought of as disclosure (like NEPA), not as a decision tool in my opinion.
    There are some deep problems with translating benefits and impacts into dollar-denominated values. High among them is the heavier weight given to benefits and impacts on the wealthy because their “value of time” (or value of whatever is being measured) is based on their income. Is it really defensible to systematically devalue impacts to the poor? But that’s built into most benefit-cost analyses.
    One more concern: many transportation agencies apply a benefit-cost assessment to specific projects, but sometimes those projects need to be in place for the benefits of a larger program to be realized. In that case, the program benefits and costs need to be evaluated, but the projects essential for program success can’t be treated the same way. There needs to be a way to account for interdependencies among projects, and to apply the analysis to the interrelated actions that need to occur to achieve the desired benefits. You wouldn’t leave out a mile of track in the middle of a transit line for example – but sometimes project-level assessments can similarly miss the point of what a larger program is trying to accomplish.
    Personally I prefer sometime more direct as a technical measure, like cost-effectiveness, to get a sense of how well money is being used to achieve some common measure of utility, like a reduction in congestion or increase in ridership. The more complicated the measure, the less likely the decision-maker will understand or use it. I agree with you that other important effects ought to be evaluated separately. The point is to figure out what’s useful to the conscientious decision-maker, not what makes the technician happy.

  13. Eric Doherty December 8, 2011 at 8:08 pm #

    Jason wrote: “I would dispute your characterization that CBA “hides” value judgements . . . those amounts are uncertain and subject to debate, but that doesn’t mean they’re hidden.”
    I have only ever come across one CBA tool where the amounts and value judgements are revealed and the spread sheets made available to the public – the VTPI’s . Because everything is transparent, it becomes rather like multiple accounts evaluation; you can apply your own values and change the quantitative values for each category. (Disclosure – I worked on the second edition).
    In contrast, most CBA reports I have come across are basically black boxes. Sometimes there is not even a list of the factors included and excluded, never mind sources and rationales for the values.
    Is it only in Canada that CBA is usually a black box procedure? Do other jurisdictions make the software and data available to the public?

  14. Morten Skou Nicolaisen December 9, 2011 at 12:26 am #

    @ Eric Doherty
    Her in Denmark we now have a standard set of transport economic unit prices, which helps but doesn’t eliminate the problem.
    First, it is a daunting task for experts, let alone laymen (i.e. most decision makers/policy makers) to go through the analyses and data that went into constructing these numbers. At best you can look at them at evaluate whether they seem reasonable. This is a unit pricing black-box.
    Second, the decision support documentation prepared by consultants often do not explicitly state the models or assumptions used to produce forecasts, making it almost impossible to evaluate their validity unless you start requesting all background reports and internal notes. Even then you might be left in the blank on some issues, such as elasticity values for induced demand (which is usually zero anyway, since it is often ignored completely). This is a unit measure black-box.

  15. December 9, 2011 at 11:54 am #

    Interestingly, today the Financial Post published a rebuttal opinion to Peter Shawn Taylor’s piece here:
    Living in Waterloo Region and having been following our own winding road towards transit improvement, I’ve read a fair bit of what PST has to say about the LRT project… none of it is good, and little of it is fair. At one point he quipped that the region should use the money to buy everyone a new car. It’s safe to say that PST has an axe to grind with this most recent article.
    I definitely took issue with his description of MAE and his approach of CBA-uber-alles. But your description really helps separate what is analysis and what is value judgement: really frames the subject nicely. Thanks!

  16. Nike High Heels 7 December 10, 2011 at 1:45 am #

    One easy-to-understand example is the ability to search, browse and quickly visualize data sets. One data scientist told us they spent significant time simply rooting around in their collection of data sets to understand what was out there, and then writing code to visualize what was in each data set. With Greenplum Chorus, that’s now an extended search function with built-in visualization — a huge time-saver.