Economist: Oops-onomics

economics

Did one of my favourite books got it wrong? Or was it just a minor flaw? Here is the criticism as reported by the Economist and Levitt's (preliminary) response here. [UPDATE(19/7/06): His final responses are here and here]

---

Dec 1st 2005
From The Economist print edition

Oops-onomics
Did Steven Levitt, author of “Freakonomics”, get his most notorious paper wrong?

ABORTION cuts crime. That claim—first demonstrated by John Donohue, of Yale Law School, and Steven Levitt, of the University of Chicago, in an academic article in 2001*—is the kind of provocative and surprising conclusion that has made Mr Levitt's book, “Freakonomics”, such a runaway success this year. Unwanted children, the story goes, are more likely to become criminals in later life. Abortion, legalised throughout the United States by the Supreme Court's Roe v Wade ruling in 1973, prevents unwanted pregnancies from becoming unwanted children. Higher abortion rates from the 1970s onwards thus help to explain why crime rates fell in America about two decades later.

That's the theory. But a paper published last week by Christopher Foote and Christopher Goetz†, two economists at the Federal Reserve Bank of Boston, finds an embarrassing hole in the evidence. Messrs Donohue and Levitt subjected the data to a battery of tests, some suggestive, others more systematic, in an effort to prove the links in the chain. The challenge is to distinguish the role of abortion from other potential influences on crime, many of which cannot be observed directly. Some of these rival factors vary year by year; others state by state. Messrs Foote and Goetz concentrate their fire on those that do both. They offer the crack epidemic, which rose and receded at different times in different places, as an example.

Messrs Donohue and Levitt claim to control for such effects in the final test of their paper. That exercise is meant to facilitate comparisons such as: did arrests of 20-year-olds in New York in 1992 diverge from those of 18-year olds in the same state and year? This automatically takes account of anything going on in the Empire state that year (such as a crack epidemic) that would have affected 18-year-olds and 20-year-olds alike. The principal difference between the two age groups is that one was born after the Supreme Court legalised abortion and the other before.

It was a good test to attempt. But Messrs Foote and Goetz have inspected the authors' computer code and found the controls missing. In other words, Messrs Donohue and Levitt did not run the test they thought they had—an “inadvertent but serious computer programming error”, according to Messrs Foote and Goetz

Fixing that error reduces the effect of abortion on arrests by about half, using the original data, and two-thirds using updated numbers. But there is more. In their flawed test, Messrs Donohue and Levitt seek to explain arrest totals (eg, the 465 Alabamans of 18 years of age arrested for violent crime in 1989), not arrest rates per head (ie, 6.6 arrests per 100,000). This is unsatisfactory, because a smaller cohort will obviously commit fewer crimes in total. Messrs Foote and Goetz, by contrast, look at arrest rates, using passable population estimates based on data from the Census Bureau, and discover that the impact of abortion on arrest rates disappears entirely. “I am simply not convinced that there is a link between abortion and crime,” Mr Foote says.

Messrs Donohue and Levitt did not run the test they thought they had

It may be asking too much of the numbers to convince everybody. “The debate over abortion and crime will not be resolved within the parameters of our paper,” says Mr Donohue. He thinks the arrest figures are “muddy” and the state population data “sloppy”. Combining the two generates so much noise, it is hard for the statistical tests to hear anything. Ted Joyce, a professor at Baruch College (part of the City University of New York), who has had his own methodological disagreements with Messrs Donohue and Levitt, also thinks the debate is stretching the data too far. He points out that if you add controls for 50 states and 12 years—as Messrs Foote and Goetz do, and as Messrs Donohue and Levitt meant to do—you are, in effect, holding another 600 things constant. This robs the data of most of their variety, and of much of their ability to explain anything.

To say, as Mr Levitt does in “Freakonomics”, that “abortion was one of the greatest crime-lowering factors in American history” may be a bit strong. But the underlying thesis, however unpalatable to some, is not likely to be dispelled by a stroke of Mr Foote's computer key. Mr Levitt says his case is based on a “collage of evidence”, of which the flawed test is one small piece. He is, in particular, sceptical that crack undermines his thesis: it varied more by age group than by state, he says, hitting 17-year-olds in all states harder than 25-year-olds in any state. He is instead trying to improve his measures of abortion, to take account of the fact that people born under one state's abortion regime might later move elsewhere to commit their crimes.

For those interested in the consequences of abortion, rather than the causes of American crime per se, the dispute might next move to Romania. In 1966 Nicolae Ceausescu, the country's dictator, banned abortion. A kind of Roe v Wade in reverse, this decision had a much bigger effect on childbearing in Romania, where women had relied heavily on termination as a form of family planning. The birth rate rose from 1.9 to 3.7 children per woman in the space of a year. A forthcoming study by Cristian Pop-Eleches, of Columbia University in New York, explores how these extra 1.8 children fared in later life. Mr Pop-Eleches offers “some suggestive evidence” that children of a given background born after the ban may have grown up to commit more crimes than those born just before—although again this may have as much to do with the changing times in which they lived.

Of course, lots of people have always thought Mr Levitt was in the wrong. Even if abortion cuts crime, it is still immoral, they fulminate. But this is largely beside the point: Mr Levitt's research does not take a position on abortion's social virtues, but aims merely to uncover its societal effects. Besides, for someone of Mr Levitt's iconoclasm and ingenuity, technical ineptitude is a much graver charge than moral turpitude. To be politically incorrect is one thing; to be simply incorrect quite another.

---

* “The Impact of Legalised Abortion on Crime”. Quarterly Journal of Economics, May 2001.

† “Testing Economic Hypotheses with State-Level Data: A comment on Donohue and Levitt (2001)”. Federal Reserve Bank of Boston working paper. November 2005. Available at www.bos.frb.org/economic/wp/wp2005/wp0515.pdf