how auditors get transit wrong: a lesson from vancouver

Elected officials love to demand "audits."  Auditing means that you hire high-prestige people who will scrutinize the books of an agency with particular genius, and deliver recommendations that resound with authority.

But many of the companies hired to audit transit agencies don't seem to understand transit.  That's certainly the evidence of a recently commissioned audit of Vancouver's TransLink, which discovered $41 million in potential annual savings including $5.3 million from cutting low-ridership services.  (Extensive detail and media reaction is gathered here.)

Like many audits, this one just assumes that low ridership means "without justfication."

But low-ridership services are unjustifiable only if ridership is their purpose.  

If you haven't read my book, or read this blog much, you may be under the impression that the goal of all public transit is high ridership, and that low-ridership services are therefore failing, evidence of waste, and should be cut.  In reality, every transit agency runs service that has a purpose other than ridership. These services, which I call "coverage" services in my own work, have purposes such as:

  • Equitably distributing service to all areas that contribute tax revenue to the agency.
  • Meeting the urgent needs of small numbers of people living in areas that are expensive to serve (seniors, disabled, isolated rural pockets of poverty etc)
  • Satisfying a coverage standard, which is a statement of the form "___% of the population live within ___ metres/feet of service".  The specific purpose of these statements, which most agencies have, is to determine when service must be operated despite predictably low ridership.

The TransLink audit appears to be simply ignorant about the universal tension between ridership goals and coverage goals.  They recommend cutting coverage services because they have (predictably) low ridership.  This is exactly as logical as throwing away your microwave because it doesn't produce ice.

When an auditor' assumes that ridership or fare revenue is the only goal of transit, they are expressing a certain set of values.  This is a valid philosophical position, but it is not the only justifiable one, nor the only one that is widely held in most urban populations.   So auditors do citizens a great disservice when they present their values as the only possible ones.  In 20 years I have never encountered a public transit agency that actually deploys service exclusively for ridership.  Now and then and auditor swoops down and criticizes the agency for the low-ridership services, often implying that the agency didn't already know about them.

Transit agencies need the backbone to reply to these audits firmly, explaining that low-ridership services may exist for purposes other than ridership, such as those listed above, and that if these services reflect the voters' values, they are as legitimate as any other.  

Transit agencies can also support clearer auditing processes if they identify which of their services are intended mainly for coverage, which means their low-ridership should never be counted as evidence of failure.  I have worked with several agencies on forming clear statements about the percentage of resources that the agency wishes to devote to coverage service.  Once those services are documented, everyone can stop complaining about the low ridership of those services, because high ridership is not their purpose.

Auditing is one of those high-prestige professions, like architecture, that is prone to form echo-chambers that resist the introduction of outside information and perspectives.  Great auditors, like great architects, are suspicious of their own echo-chamber and always looking for perspectives from outside of it.  If you want to be a good auditor of public transit agencies, read my book!  It will help you avoid the TransLink auditor's mistake, and many others.  

 

29 Responses to how auditors get transit wrong: a lesson from vancouver

  1. al m October 23, 2012 at 6:18 pm #

    You hit the nail on the head..
    It’s NOT all about the MONEY!

  2. Erik H. October 23, 2012 at 7:45 pm #

    Sounds like S.O.P. for TriMet – focus on MAX and other rail services; bus service is only a “nice to have” and is the first to get cut – yet it’s the bus system that ensures fair and equal service to everyone living in the service district.
    Thanks to multiple rounds of service cuts, many riders now have a two or three transfer bus ride, who used to enjoy a single-seat trip. (But not a MAX rider from Gresham to Hillsboro.) Many of these outlying bus routes now use older, less reliable buses; meanwhile fares have gone up, and transfers are inconvenient – what used to be a one minute stop has turned into a 20 minute wait (and on Sundays up to a 60 minute wait) – surely designed to discourage ridership, rather than encourage it.

  3. rico October 23, 2012 at 8:29 pm #

    I have not really looked into which routes were cut but previous to the bad times Translink increased the frequency of a lot routes in places like Surrey in an effort (that likely would have succeeded) to boost ridership….so those routes may be the services being cut. That said I believe the vast majority of ‘savings’ the audit found were excessive contingency allowances…so not actual savings except on paper. Like a lot of other places Translink needs more funding and politics is getting in the way of getting it.

  4. Bruce Nourish October 23, 2012 at 9:07 pm #

    More generally, even for a route not set aside as a “coverage” route, it’s very dicey to think you can fully understand the performance of a route by just looking at route-level statistics. For example, King County Metro’s Route 26 performs great for half its length, then falls off a cliff:
    http://seattletransitblog.com/wp-content/uploads/2011/11/king_county_metro_26.png
    The section on Latona has some coverage value (it’s about 1/2 to 3/4 of a mile from Route 16, its nearest neighbor), but it may not warrant the one-seat ride downtown off-peak; especially if cashing in the one-seat ride allows you to make a very advantageous restructure in a part of the network nearby (which it happens to in this case).
    But you’ll never figure any of this out looking at the route-level stats.
    Perhaps, as a best practice, auditors should be told that they can’t evaluate route performance, and instead hire a consultant to evaluate their network design on its merits? Have the auditors focus what they’re supposed to be good at, namely finding genuine financial inconsistencies or abuse.

  5. leccy October 23, 2012 at 9:38 pm #

    Evening services are very interesting. They can often have relatively low ridership but if you cut them there are flow on effects whereby people can no longer rely on the route as providing an ‘all day’ service. This means a switch to car, not only for the night but also for the daytime and peak services as well.

  6. Steven Jamieson October 23, 2012 at 10:04 pm #

    This is especially relevant to us in Brisbane/South-East Queensland at the moment (the other, other TransLink):
    http://jp.translink.com.au/travel-information/service-updates/details/1346370543

  7. Rob October 23, 2012 at 10:38 pm #

    The word “performance” in performance audit seems to open the door for an auditor to substitute his or her own values for those of an agency or elected leaders. I saw the same issue raised when an elected state auditor found that a state department of transportation should have had a stronger mission to eliminate congestion, despite legislative goals that balance congestion relief against other social values. It seems to be a ticket sometimes to rewrite the rules without accountability – something that seems counter to the purpose of auditing! Auditors need to be accountable too – and that should start with a commitment to measure an agencies’ achievement of their own mission, not the mission they would have if they shared the auditor’s personal values.

  8. Eric2 October 24, 2012 at 2:02 am #

    “They can often have relatively low ridership but if you cut them there are flow on effects whereby people can no longer rely on the route as providing an ‘all day’ service. ”
    Not just evening services. The existence of service to peripheral areas increases ridership on the core routes, by peripheral residents who transfer to the core routes, and by core residents who know that it is possible to get to peripheral areas if it ever becomes necessary.

  9. Eric2 October 24, 2012 at 2:05 am #

    By the way, I would submit that a late night route is typically faster (less traffic) so uses less driver time and expense to cover a given route. If both ridership and route time are halved, the overall expense per rider is not much higher.

  10. John Levin October 24, 2012 at 4:22 am #

    While I agree with the general idea that ridership and productivity should not be the only measures of transit success, there does need to be a lower bound to performance.
    Providers define different performance standards by service type to address this, with lower standard for route types that have other benefits such as coverage.
    But in the end, when routes don’t meet those performance standards, they need to be reconsidered. I would argue that we do need, at some point, an objectve performance threshold by which to measure our routes.
    While the sound bites of the TransLink audit suggest an aggressive, and less nuanced, approach to this issue, it is not clear to me, in reading the document, where the line is being drawn.
    What is the performance of the routes that would be proposed for cuts? Where does that fall in the terms of standards and/or reasonable performance? [rhetorical questions….]

  11. MaxUtil October 24, 2012 at 9:36 am #

    A basic rule of the auditing profession is that you must find a “waste” or “potential savings” that are some multiple of the fee that was paid for the auditing service. That’s just good business. No one hires consultants to hear “No, they’re doing a pretty good job.”

  12. david vartanoff October 24, 2012 at 10:22 am #

    Good post, Jarrett. We need to remember that the capillaries are necessary to the arteries and veins. As others have said, but deserves repeating, the perception of reliable (translates to when I need to travel) coverage supports base day and rush hour usage. Auditors should spend their time looking at MDBF, AWOL, OT, and other actual statistics indicatibve of systemic failure.

  13. Alan Robinson October 24, 2012 at 11:56 am #

    @ MaxUtil
    Case in point from the last point in the executive summary of the audit:
    “Throughout the review, the Steering Committee observed an overarching emphasis by Translink in its business decisions and culture to focus on customer service. While this is a critical area of focus for any business, for those that are publically funded and to respect the taxpayer contribution, a balance must also be sought between service for users and efficiency of operations”
    The province really can’t say that Translink is doing a superb job.
    Although I’m not back home to personally gauge the public perception, I definitely get the feeling that this particular attack by the province on the revenue and governance of Translink is not going anywhere.

  14. Andre L. October 24, 2012 at 12:42 pm #

    I think in many cases transit audits are hired because a new government/director/comptroller wants to have some accountability and, being no better or worse than any other government agency, the employees of transit agency have all self-preservation and turf-guarding instincts to not change what THEY think it is the right thing to do – considering they are the “specialists”.
    So an external audit is positive in that it can “turn up the heat” and expose numbers that might have a reason to be as they was, but then force the agency to go two extra miles explaining its policies and justifying things like low-ridership services.
    If we couple Jarret’s idea on this post with the other one where he says he doesn’t like to provide route-specific data to the public or even to their bosses because they wouldn’t understand what lies behind the numbers, a very worrying picture emerges, where someone is proposing less transparency under the pretenses “people can’t understand numbers except ourselves working on this business”.
    Looks a recipe for financial disaster.

  15. SLB8988 October 24, 2012 at 2:15 pm #

    I find it hard to beleive that auditors are unaware of the methodology behind transit planning as a public service… They are likely VERY familiar with the way we do business. I don’t mean to offend anybody here, but have we ever considered that they way we “do” transit isn’t the most effective and that it ultimately works against us in the public eye? Yes, running a route out to a poor rural activity center makes you feel good, and its satisfying a real need, but it also works against EVERY OTHER benefit that transit provides to the community – economic development, density, urbanism, walkability – and it supports existing disconnectivity in cities and urban sprawl. Rural poverty is a land use problem and affordable housing needs to be provided in urban areas. As much as I hate to say it, the majority of the American public – including TD populations – view transit as a “social service” with a negative stigma. Ridership is MORE than just money, it is SUPPORT. And from a national perspective, transit NEEDS more support and attention from elected officials and from the public. Why do transit $ referendums fail across the country? Because people don’t think they’ll ever use the system. We need to show the public that transit does work and works very well in urbanized areas, but that will never happen if we continue to structure it like a social service and keep sacrificing efficient service in the city for rural service. The argument that service needs to be spread across an entire jurisduction for the sake of tax equity is simply ridiculous – its like building a school in a retirement community just because the school board collects taxes from elderly residents. Transit needs to go where it can be supported. Until we accept this and change the way we provide and plan for transit, we will never get more folks on board in this country.

  16. Bruce Nourish October 24, 2012 at 2:52 pm #

    “[Jarret] doesn’t like to provide route-specific data to the public or even to their bosses because they wouldn’t understand what lies behind the numbers”
    Setting aside whether that’s true, route level statistics alone can be very misleading to *anyone*, as my example above shows. That alone is reason not to publish them. Uninformed debates are better than authoritatively confused debates.

  17. Aleks October 24, 2012 at 6:07 pm #

    This discussion has made me wonder if it’s actually possible to meaningfully distinguish productivity-based services from coverage-based services, given the importance of network effects.
    Sound Transit runs a route, the 545, that goes between Seattle and Microsoft’s campus in Redmond. Needless to say, this route is much more productive on weekdays than weekends. In fact, the productivity of the Sunday service is rated as marginal.
    Should Sunday 545 service count from the ridership budget or the coverage budget? On the one hand, providing a high-quality baseline service, 16 hours a day and 7 days a week, is clearly a good thing for ridership. On the other hand, the Sunday service’s poor performance suggests that it exists mostly to meet a frequency/span standard, and so its existence cannot be justified by productivity alone.
    Now, it’s certainly true that some routes are clearly explicitly designed to meet coverage goals. Routes like King County Metro’s 249 are so circuitous that no amount of investment in frequency, span, or speed could ever make them productive. But still, I’m imagining a citizen of Sparseville taking a look at a proposed service plan, and asking why we can justify running near-empty Sunday 545s out of the “ridership” budget, but relegating the weekday 249 to the “coverage” budget (presumably with a lower standard of service).
    I wonder if it would make more sense to present the relationship as a graph of ridership versus productivity, where the curve represents a given level of budget. At the upper left, you show the maximum ridership, along with (probably) the minimum coverage area. At the lower right, you show 100% coverage, along with (probably) the minimum ridership. That way, you still convey the nature of the resource-allocation problem, but you do so in a way that doesn’t imply that any given service is exclusively ridership-oriented or coverage-oriented. And it’s possible to evaluate the network against both goals, with the knowledge that both goals are realistic to achieve given the budget you have.

  18. Aleks October 24, 2012 at 6:20 pm #

    In general, I do think that government agencies have an obligation to provide any data they have to the general public, even when it’s not pleasant to do so.
    However, that extends only to raw data. Agencies are under no obligation to aggregate the data in any particular way. And, in general, it’s questionable whether aggregating productivity data on a route-level basis is a useful thing to do in the first place. There are network effects, and time-of-day effects, and bipolar effects (where a route that combines a highly productive base with a minimally productive tail looks like a middling route on average).
    We know from first principles that, when density increases, ridership increases faster. Therefore, the question that we really want to ask is, for a given city area, are we seeing the kind of ridership we expect?
    Imagine that, for each census tract in a city, you took the total number of transit on/offs (for some time period) and divided it by the population-weighted density. Then you drew this as a heatmap, perhaps normalized against some other city of comparable size. You would very quickly see which areas of the city were getting the kind of transit ridership they could, and which areas were falling behind. That would give you a pretty good indication of where improvements would be most needed.
    In summary, there’s no need to impose any kind of government secrecy. It’s just that aggregating productivity by route is misleading for everyone, and there’s no reason to calculate those numbers in the first place.

  19. Rob October 24, 2012 at 6:56 pm #

    I think a lot of low-productivity tails are chasing hard-to-locate turnaround and layover spots. An auditor that wants to dig into the route-and-segment level weeds will need to understand what are the real alternatives that drove route design choices.

  20. Brent October 24, 2012 at 9:09 pm #

    KPMG was hired by Toronto during last year’s budget debate to do a major audit of the city’s expenses (and one that was highly politically charged).
    One of their recommendations related to the overnight bus service (the “Blue Night Network”, running a skeleton service mostly every half hour from 1-5 AM). They suggested that the fare be raised (or doubled) on those overnight routes.
    This struck me as being backwards — usually if fares vary by time of day, they are higher in the peak periods to try and encourage ridership during the off-peaks when there is spare capacity and spare vehicles available.

  21. Aleks October 24, 2012 at 10:38 pm #

    @Rob: That may be true in your city, but in Seattle, we have a long history of providing radial service between downtown and every minor arterial that calls itself a neighborhood. For the route that Bruce cites (the 26), over 1/3 of the mileage of the route is the tail.
    Fremont Ave between 34th and 39th is particularly vexing for someone who likes simplicity. Six separate routes pass through that stretch — the 5, 26, 28, 31, 32, and 40. Not one of them goes straight. With all the redundant routes we have, we could easily afford to run non-overlapping routes with 10-minute frequency on all the important corridors. Instead, we have chaos, and useless long tails.
    @Brent: There’s two general approaches to fares. One is to use demand-driven pricing, and charge rates based on the amount of free space you have (or want to have). The other is to use cost-driven pricing, and charge based on the average cost per rider of providing the service.
    I believe that the former is better for many reasons, not least of which is that it can end up saving money in the long run by leveling out the peaks and encouraging the use of excess capacity. But from the perspective of an auditor who knows little about transit, it’s hard not to look at a near-empty bus with poor farebox recovery and ask, “Why aren’t the riders paying their fair share?”.

  22. John Smith October 25, 2012 at 2:39 am #

    An interesting and wide ranging debate.
    At least one contributor has pointed out that the real issue isn’t public transport, or ‘accessibility’ as we currently call availability of services here in the UK, but land use.
    I live in North East England. We have numerous small settlements spread across County Durham that were previously based on coal mining. The mines have long gone, but the villages hang on. Being (relatively) remote from larger towns and cities, they need schools, medical facilities, retail and, of course, public transport, even though the primary purpose of the settlement has disappeared. In the 1960s and 1970s, a progressive County administration faced up to the problem and identified what it called ‘category D’ settlements that would be allowed to ‘Die.’ There was an enormous backlash from residents of such places, because they were settled in places in which many of them had been born, grown up and had their own families.
    Public transport arose to respond to expanding urban populations, then got the technology to take the service beyond to neighbouring towns and villages. When the purpose of the village disappears, then surely it’s time to re-evaluate the settlement? When petrol (sorry, ‘gas’) was cheap, there wasn’t a problem. That there is a problem (and that it’s a problem that public transport can’t solve on its own) is something that the next generation of politicians – across the developed world – should be prepared to face up to.

  23. Andre L. October 25, 2012 at 2:56 am #

    @Bruce: this is what Jarret wrote on the post

      perils of “fare revenue by route”

    :
    If you have financial managers who don’t have transit in their bones, they can easily fall into the illusion that the routes are like independent products […] and this can lead to some poor decisions.
    If the managers really understand interconnectedness in transport networks, then they may find the info useful, but I am still reluctant to prepare data outputs that are likely to be published without their explanations, and this one can be very misleading if you dont’ bring that element along.

    This is why I meant when I wrote that Jarret is advocating less transparency, or stalling some aggregated reporting required by transit bosses/politicians, just because data could me “misinterpreted”
    ===========================
    @John Smith: I think you are twisting the reasoning – we need to develop new technologies that would allow sparse settlements to thrive and be connected to bigger cities, not slate cities and towns to die, assuming never better than a “diesel-run, human-driven bus” will ever come to place.

  24. John Smith October 25, 2012 at 6:24 am #

    @Andre L.: No-one moves to a remote rural settlement for work – it’s almost always a lifestyle decision. So why should the rest of society be obligated to pick up the bill for providing education, healthcare and transport for people who choose to move to such places? If a community becomes ‘unsustainable’ (and not just in the ‘green’ terms with which this phrase has become associated), then why should we attempt to ‘sustain’ it?

  25. Alon Levy October 25, 2012 at 8:49 pm #

    Jarrett, I’m surprised. The auditors’ report regarding the savings from eliminating marginal coverage routes is useful information for people who want to decide whether they value coverage routes more than they value additional service on ridership routes, diversion of money to other spending priorities, or lower overall government spending.
    If you want people to be able to express their values, make sure they have all the information they want or need. If the total savings from eliminating coverage routes are really just $5.3 million a year then to me it’s no big deal relative to Translink’s overall budget and the routes should probably be retained. But if those savings were higher, I’d think differently, depending on how much higher.

  26. Tex-Mex October 28, 2012 at 7:08 am #

    I once sat through one of these audits where a four-letter named audit company, in the course of a transit audit, said that our goal should be to focus on the third of bus routes with the lowest productivity which, in their measure, meant a rote ranking of passengers per revenue mile of service. I asked them when we cut all express routes and all of the services to some very low income areas on the fringe, what happens next service change? Do we again focus on the lowest third of the remaining routes? Obviously without thinking, they agreed that this was the strategy. A few years later, rounding of the math would presumably allow us to keep the last remaining bus route, or maybe two, under their approach.

  27. MB November 2, 2012 at 12:53 pm #

    The TransLink audit was commissioned by the government of British Columbia and in my view is mostly about politics.
    The audit was conducted in the context of years of chronic underfunding by senior governments to a metropolitan transportation agency with purposefully limited taxation and decision-making powers. They are trying to make the lack of funding look like it’s TransLnik’s fault. This has created an untenable situation in light of high growth in the region.
    The same government is concurrently completing a multi-billion dollar freeway system (probably in the range of $9 billion with financing costs over 35 years) including “the world’s widest bridge” (yawn), so let’s be clear about the priorities of the party that hired the auditors.
    TransLink was created by the BC government under a left-leaning administration in the 90s, a few years before the current free-enterprise party took office. It was initiated to assuage justified concerns about local control over transportation.
    TransLink has on occasion been a thorn in the side of the current government when their decisions did not dovetail with the government’s wishes, and as the result the government monkeyed with TransLink’s governing structure, taking real decision-making powers away from elected civic politians and giving it to an appointed board. A new, toothless Mayors Council was created as a sop to “local control”, but their role is now only advisory. It’s a joke. (I can say that, Jarrett, because I’m neither employed or likely to be employed by TransLink.)
    The best way to rectify this comedy is, in my view, to legislate a directly-elected Metro Vancouver regional government consisting of a board of directors, many of whom would also sit on the boards of sub-governing bodies like TransLink. An elected regional government would have powers of taxation, but these will have to be harmonized with the individual municipalities and senior government to avoid duplication. I would also throw in land use planning powers to ensure the investments in transit are backed by transit-oriented development, hopefully designed at the human scale.
    Stephen Rees, a retired transportation economist and former consultant with TransLink also posted his interesting take on the audit on his blog:
    http://stephenrees.wordpress.com/2012/10/18/translink-audit-complete/

  28. daodao November 3, 2012 at 12:50 pm #

    In the UK, many rural English counties are withdrawing subsidies for nearly all evening and Sunday bus services, leading to their withdrawal, as they are not commercially viable. Such changes were implemented in Shropshire in 2011 and are due to be implemented in Cheshire East from January 2013. Bus companies will not voluntarily run non-profitable services, as they would go bankrupt. Surely the same capitalist principle applies in the USA, where provision of optional services by government is considered an anathema?

  29. bloxorz online August 10, 2017 at 12:59 am #

    A great blog, I will share it with my friends, thanks for your valuable posts