In the last post, I noted a ranking of the "10 best US cities for transit" in U.S. News and World Report back in February, and some incoherence in how the ranking was explained. Since then, I've become even more disturbed by the rankings.
It turns out that Michael Andersen of Portland Afoot had done some research, or attempted to:
I was pretty interested in this, in part because Portland's TriMet is, understandably, promoting the ranking heavily. I thought it was great news and wanted to include more details in our magazine.
So I called [Danielle] Kurtzleben, the reporter who'd compiled it, to ask about her data sources and methodology. After five emails/tweets/phone calls over several weeks, I finally got her on the phone, at which point she said she couldn't remember exactly how she'd figured the data, except that it came from APTA and the NTD and that it was "very simple." I asked if she could email me the spreadsheet; she referred me to her editor, who said, bewilderingly, that U.S. News policy is to not share the data it gathers.
So I tried to retrace Kurtzleben's steps. Here's the result; my summary is at the bottom of that page. I pulled the data apart six ways from Sunday, based on the somewhat sketchy description in her article, but couldn't come up with any scenario that ranked anybody above New York City, whose ridership and funding ranks dwarf all others.
Her three metrics were total spending per capita, boarding-rides per capita and safety incidents per boarding-ride. It's not clear what types of "safety incident" counted or how many years of them she analyzed; how she weighted the three metrics into a single ranking. I also suspect she may not have noticed or considered that population data in the APTA handbook is based on population figures from 2000 — the only place to get apples-to-apples population figures is the ACS [American Community Survey], which she didn't mention using. But even after I ran several variations using the 2000 figures I couldn't duplicate her findings.
Another possibility is that she could have failed to fully account for all the spending and ridership at metro areas that have multiple transit agencies; Portland's relative lack of overlapping suburban agencies would help explain its good ranking. Or she might have calculated population by city rather than metro area.
At any rate, I think I made a good-faith effort to explain these numbers and couldn't.
My one-sentence summary: This article cited out-of-date population figures and was calculated with a methodology that U.S. News refused to explain, based on figures that U.S. News refused to share.
This is a little distressing, especially for a study that's being widely cited by the transit agencies in question. If you know anyone who might be able to confirm that the rankings are based on, well, something, or anyone else who's tried to do a similar analysis, please send them a link.
Bravo to Michael for expending all this effort in the search of reality. I don't know if he's right, but he certainly deserves an answer.
More on this topic here!
So basically this ranking is just as valid as their college rankings?
Because newspapers are newspapers, they think they don’t need to prove that anything they say has any merit; they are too arrogant to provide sources or methodology, unlike scientific journals or even blogs.
Is that the same US News and World Report that makes the university rankings? Because that is another ranking that 1) sketchy at face value and 2) getting more attention that it should be
Also worth checking
http://en.wikipedia.org/wiki/U.S._News_%26_World_Report#Criticism_of_college_rankings
A completely citation-proof article citing all its sources, I think this one in particular is relevant
http://www.educationsector.org/usr_doc/CollegeRankingsReformed.pdf
1) I find it ironic that journalists critisise wikipedia, then use phrases like “experts say”, without saying who those experts are.
2) Jarrett, how would go about ranking transit agencies in North America?
I did not realize anyone took any U.S. News rankings seriously. They try to rank everything and everyone so that the people who have been ranked turn around and tell everyone they got high rankings and those people check out U.S. News.
I was always disgusted that my small college would use U.S. news materials saying they were “ranked as Americas Top Colleges” When they were ranked ~75 for Western Region Online Master’s Program. U.S. News to me has always been a scam.
If the ranking is in fact inaccurate, it’s especially harmful to the extent that it reinforces TriMet’s conviction (and UTA’s, etc.) that they’ve been doing everything right.
I tend to sympathize with most of TriMet’s strategies, but in a city as prone to self-mythologizing as Portland, false validation makes it hard to maintain a healthy skepticism. In my opinion, groundless optimism is a serious operational problem for TriMet.
Anybody that knows what a bus is could take a look at that list and know it isnt worth cleaning up spilled coffee with.
Rankings are obviously subjective, but to put SLC or Portland above NYC in a transit list is simply idiotic.
Subjective would be deciding if SLC is better than Portland, ie, which one gets 6th or 7th place.
The difference between this and the college rankings is that here they got caught. With the college rankings, at least they have common wisdom as a reality check, so they’ll tweak the methodology to make sure Harvard, Princeton, and Yale are at the top. With transit, there’s much less of this, so it’s easy to make a mistake and think Portland and SLC could be the top cities.
A ranking that started from mode share and broke near-ties with investment levels and direction of mode share would look like:
1. New York
2-5. left blank for illustrative purposes
6. Washington
7. Chicago
8. Bay Area
9. Boston
10. Philadelphia
11. Los Angeles
12. Portland
13. Baltimore
…
666. Dallas
…
9001. Austin
Actually, Portland fares well on Alon’s list, as the cities ahead of it are all far larger. That said, one could make a good argument for Honolulu above Portland, although (as is usually the case) many of the “facts of the ground” (very high residential density, higher-than-average fuel prices, and lots of lower-income residents) make TheBus a more compelling option for more people–but Honolulu achieves excellent transit share (higher per-capita than Portland) with only a bus system. (Rail is in the works, but isn’t running yet).
Well said, Alon.
@ Engineer Scotty, your description of Honolulu echos Joel Garreau’s analysis of the HK Metro decades ago, take the Subway , swim, or live in the PRC (why it was the ONLY Metro showing a profit)
I was under the impression that the major emphasis was on transit investment and safety records, adjusted for population so that big population places wouldn’t dominate too greatly.
Unfortunately, it seems with the research done by Portland Afoot that there were no such complex calculations. Metro Transit in the Twin Cities has been boasting their ranking, and as much as I utilize transit here and love the cities, I don’t think it ranks #5 in the nation. As others have pointed out, their rankings are suspect.
I think a more compelling list, which I am trying to work on, would evaluate how easy it is to get around each city without a car. I think such a rating would have to be subjective, but I would be open to comments about how to measure objectively the ease of non-automotive urban mobility.
One of the problems with ranking city transit systems is that any transit system varies widely in quality between the central city and the distant suburbs. “Bay area”, for example, encompasses everything from arguably the 2nd best city to get around in without a car (San Francisco) to an area where if you don’t have a car you are literally trapped in your home (County Connection service area).
To this day I don’t understand why people believe Portland has a better transit system than Seattle. Please enlighten me.
@Alon
You could fill 2-5 with Montreal, Mexico City, Vancouver, Toronto, and make it a North American ranking 😉
Ant6n. In transit or urbanist writing "North America" is usually a term economic and cultural geography rather than natural geography, so that usage excludes Mexico and Central America.
(And yes, this "North America" is economic more than cultural, so it can include Québec without questioning the distinctiveness of its culture! What really matters to transit is the cost of labor.)
If you need authority for this definition "North America," see the Spanish word norteamericano, which refers to the same area.
Emm, didn’t really want to start a “What’s North America?” Discussion. Just wen’t by this List of North American rapid transit systems by ridership.
But instead of Mexico City, one could put in Ottawa or Calgary.
Jarrett, I dont think those who exclude mexico and central american from “north america” are thinking about anything. Ask them to name every country in geographic north america and they’ll usually be around 35 short. It’s ignorance.
And we can always blame it on a conspiracy theory to exclude Mexico City because their metro blows away almost everything else on the continent, and that’s just embarrassing for the richer countries to be “beaten” by a poor country with drug problems.
Sort of like excluding Moscow in talking about transit systems in Europe. Can’t have the commies win.
Including Canada on the ranking in my above comment would open the following big can of worms: the two cities with the most mode-share increase, Vancouver and Calgary, rank 4th and 5th by current mode share.
It’s for the same reason that I cut the ranking below Baltimore – though David’s right that I should have included Honolulu, which should be between LA and Philadelphia. I did not want to start comparing Denver and Salt Lake City to older cities with higher but stagnant mode share.
P.S. Off-topic, can we let go of the myth that only the Hong Kong MTR pays for itself? The Helsinki Metro is profitable, and so are most major Asian subways: Delhi, Tokyo, Osaka, Singapore, Taipei, Shanghai. Seoul has some profitable and some barely money-losing lines. The only large money-losing subway in Asia is Beijing, and that’s due to a political decision to keep fares low to fight air pollution.
The “Ten Best” articles are a marketing ploy to build awareness of the puplication to then build sales. It’s easy on the brain to read and readers are drawn to them to see if they’re lucky or smart.
huffingtonpost.com lists them in their right column that holds the gossipy, yellow journalism stuff.
The administration at Reed College (coincidently in Portland, OR) refuse to participate because they think the criteria is bogus.
I’ve thought for years the “top ten” lists were useless but last year US News had a Ten Best Places to Live article. They listed my hometown of Baldwin, NY on Long Island. It was a joke.
Chris, many people believe that Portland has better transit than Seattle because they dislike buses, and Seattle took forever to build any serious rail transit. (By serious I mean that they had the monorail and waterfront trolley, but both are only used by tourists, or people who ride the bus so rarely they don’t realize it’s faster than either of those.)
Steve got it.
The U.S. News “best” lists stemmed from the attempt to shore up dwindling readership. It hit pay dirt with the college rankings.
It didn’t necessarily succeed with prospective college students or parents, who rarely cite the lists as a guide to inform their choices.
What had happened, though, was that the college list took a life of its own among the higher education institutions. College executives, fundraising offices and department chairs gave the rankings tremendous authority, and are guided by it to try to boost their rankings.
Nowadays, though, anybody and everybody knows a numbered list is a good way to get attention and start a good argument. An even lazier way to do that is just to use the words “overrated” or “underrated”.
I have my own contributions to pseudo-authoritative transit rankings.
On MetroRiderLA, I liked to simulate major sports playoffs by taking the actual team brackets and substituting the teams’ respective transit systems in them.
Sometimes, the actual teams and the transit systems performed identically.
For instance, in the 2009 World Series, both the Yankees and Phillies and MTA and SEPTA made it into the series. Yankees beat the Phillies, and well, in the World Series (of Transit), MTA won by forfeit because SEPTA went on strike.
What did I use for the playoff methodology?
For the transit-related components (in case of a tie, I would use actual sport-related stats), I did factor in quantitative factors for transit.
The games were divided into bus, urban rail, mainline rail (commuter rail and Amtrak) and transit systems’ official homepages.
For the bus and rail categories, I considered all services available within one-half mile of the sporting venue. I gave each venue a score based on number of bus/rail lines, their respective frequencies and span of service.
For mainline rail, I just considered the presence of services within the metropolitan area; it didn’t have to be within proximity of the venue. If it was, as it is in Boston and New York, it would be a bonus.
For the websites, it was more subjective, but I weighed factors such as plain-text schedules (use of PDFs would subtract points), ease of finding information (how many pages it took to find venue data), ease of use for new riders and finally, overall look and feel. I came to the conclusion that Boston’s MBTA.com and Portland’s TriMet.org pretty much set the standard for what transit websites should be and would automatically give them the win.
This was a competition, to see which transit system was better. But like an athletic event, each team must have a sporting chance to win. New York or Boston would win every series it was in, so there had to be a way for other teams to stand a chance against them.
This is also why I waited until the playoffs. These teams could have a bad year and not appear.
The sporting chance did allow some interesting dynamics. For instance, Boston’s bus service is surprisingly poor around Fenway and the Garden. The routes are not frequent, end early, and have very short distances. Boston usually drops a bus game and dominates urban rail, mainline rail and the website.
Also, there’s a chance for an overall lousy transit system to win if they can have collect several routes near a venue. I’ve given Cleveland, Charlotte and Cincinnati wins because of this rule. Those cities have routes that have 30-60 minute service almost everywhere, but their downtown transit centers are within walking distance of their venues and their services run late enough to plausibly take a bus home from a night game. These cities can make up for their deficient individual services in volume and still win.
Interesting: Another TOP 10 was published by The Atlantic on May 17th. Regards, Paul in Prague
http://www.theatlantic.com/business/archive/2011/05/the-10-best-and-10-worst-cities-for-public-transportation/238985/#slide1
Thanks to this discussion, I assume, Portland’s #1 ranking has now been retracted by US News. The correction is dated May 17, three months after the US News ranking was first published and a few weeks after this exchange.
Not exactly a happy ending, but as a reporter I’m gratified.