How To Lie With Statistics

So I happened to see an eyebrow-raising headline the other day at USA Today: North Dakota tops analysis of corruption.

This smelled a little fishy to me. More corrupt than Illinois? More corrupt than Louisiana? More corrupt than New Jersey, for cryin' out loud?

Well, that's what the story said:

On a per-capita basis, however, Illinois ranks 18th for the number of public corruption convictions the federal government has won from 1998 through 2007, according to a USA TODAY analysis of Department of Justice statistics.

Ay, therein lies the rub. This is a case where, as we move from the original statistics to the journalistic analysis to the news story, that the truth gets increasingly buried. I'm reminded of the old dictum--attributed to Mark Twain--of there being three kinds of untruths: lies, damned lies, and statistics.

The Justice Department statistics list the number of convictions for public corruption that have been made in the federal courts, broken down by state.

And it turns out that if you take each of these numbers and divide them by the number of people in the appropriate states, North Dakota has the highest number of federal public corruption convictions per capita of any state in the nation.

I suppose it's inevitable that the headlines would start trumpeting that North Dakota is the most corrupt state! But that conclusion does not exactly follow from the evidence they present.

Here are some potential ways their conclusion breaks down:

• This metric only counts the number of convictions, not the number of crimes. You only get a conviction when the person is caught. If no one ever gets caught, you can have all the corruption in the world, and still have a low number of convictions--and thus a lower ranking than everyone else. It may be the case that North Dakota is just better at finding its crooks than everyone else--resulting in more convictions, and thus a higher ranking on the list.
• This list only counts federal public corruption convictions. What about state public corruption convictions? It could be that other states catch their own crooks and try them in the state court systems, and North Dakota just happens to be deficient in this regard, so the feds are the ones doing all the work there. It doesn't automatically mean that North Dakota actually has more crimes going on.
• The statistic is counted as a per-capita with respect to the population of the state, not with respect to the population of the state's public servants.

Here's what I mean by this last one: California has 120 people in its legislature--40 in the senate, 80 in the assembly. But California has 36.5 million people. Now, suppose that--on average--one crime is committed per year by each member of the legislature. (I don't know the true figure here; I'm just assuming that they're all crooks, for the purpose of this analysis.) :-) That means 120 crimes are committed by the legislature, or 3.3 crimes for every million people in the state.

North Dakota, on the other hand, has 141 people in its legislature--47 in the senate, and 94 in the assembly. But North Dakota has not quite 640,000 people. If the legislators of North Dakota were just as corrupt as that of California (in our example), there would be one crime per legislator per year, or 141 crimes committed by the legislature, for a total of 220 crimes for every million people in the state.

With the probing analysis of the USA Today headline writers, they'd conclude that North Dakotans are (220/3.3=) 67 times as corrupt as Californians--whereas the truth is merely that the legislators of each state are equally corrupt, but North Dakota just has more corrupt legislators per capita.


...


To be fair, the story does walk back its headline somewhat, and it gives the North Dakota partisans the opportunity to give their opinions as to why the reporters are full of it:

Don Morrison, executive director of the non-partisan North Dakota Center for the Public Good, said it may be that North Dakotans are better at rooting out corruption when it occurs.

"Being a sparsely populated state, people know each other," he said. "We know our elected officials and so certainly to do what the governor of Illinois did is much more difficult here."

And later:

The analysis does not include corruption cases handled by state law enforcement and it considers only convictions. Corruption may run more rampant in some states but go undetected.

Yeah, yeah. But the trouble is, when you have a spectacular headline like this, that some non-trivial percentage of people will read the headline, and get the meme North Dakota is the most crooked state of the Union! in their heads--without bothering to read through to the end where all the walk-backs are, and without bothering to think about whether the evidence presented in the story actually adds up to the conclusion they're trying to push in the headline.


...


My wife just asked the question: "So were the reporters from Illinois?" I'm shocked, shocked that she could be so cynical.... :-)

But then, her mom grew up watching the political scene in Kentucky....