How Science Works

National Public Radio ran a brief story on Friday, July 22, about an article in the prestigious journal Science.  It was originally published in 2010, and claimed that the likelihood that someone lives to be 100 years old could be predicted with 77% accuracy based on 150 genetic markers.  The study was subsequently retracted last week.  The authors had apparently realized there were mistakes made in how the data were collected.  The error was not caught during the peer-review process.

What, you might ask, is the peer-review process?

The final, critical step in research is publishing the results in a scientific journal to add this small piece of knowledge to the collective whole.  The researcher writes up the paper and submits it to a journal, whose editor decides whether the paper fits the journal and is good enough to be worth further review.  If the manuscript passes first muster, it gets sent out to two or three people who are thought to be competent in the subject area, scientific peers of the manuscript's author.  These people are asked to review the paper and decide whether the methods are defensible, the conclusions reasonable, and the overall manuscript worthy of publication.  The editor makes the final decision based on the reviewers' recommendations and the editor's own review.

NPR zeroed in on the fact that this paper had come under criticism for its methods, and they interviewed one person in the field who felt the paper's flaws should have earned it a double thumbs down during the review process.  They didn't interview the original reviewers or the editor who originally decided the paper should be published.  NPR also didn't tell their listeners that a small percentage of papers in the scientific literature are retracted every year, as flaws in experimental design or data analysis are discovered that negate the results. 

Even scientists make mistakes.

The major story here was missed entirely: the system works, more or less, which is about as good as any other human endeavor.  Science is an iterative process, and although redoing someone else's work never earns the accolades that the original work received, the repeated testing is crucial.  This often gets overlooked, as Science (the journal, not the discipline) and other top-ranked publications vie to publish the most cutting-edge, and therefore the least-tested, results.  This tendency inevitably leads to the occasional retraction.

Science (the discipline, not the journal) advances as people come up with ideas, and then disprove them or fail to do so.  Bad ideas may arise from mistakes in data collection, data analysis, or simply from lack of sufficient knowledge to describe a system or pinpoint an underlying process.  The crucial part is not that someone got it wrong, but that the mistake was found and ultimately corrected, thus advancing what we know.

An example can be found in the early efforts to describe the structure of an atom.  This is a pretty weird idea, one that currently involves mostly empty space, a tiny and impossibly dense nucleus, and electrons whizzing around in set orbits of very specific shapes, moving so quickly they are equally likely at any point in time to be anywhere within this orbital shell.  It is such a weird idea that chemists resort to ball and stick models to describe structures made up of multiple atoms even though we know that is not at all what an atom or molecule actually looks like. 

P orbitals, or where one series of electrons in an atom exist. 

Illustration from National Institute of Standards and Technology

Ball and stick model of the sugar glucose.  Image from NASA.

This wasn't always the accepted view.  The physicist J.J. Thomson suggested based on his early experiments that the atom was more like a plum pudding, with the plums representing the electrons that were embedded in the atom.  A decade later, physicists had to admit that this idea was wrong based on work conducted both by Thomson himself and by Ernest Rutherford, who had been Thomson's student.  It seems rather silly in hindsight, but Thompson's early experimental data supported his pudding model.  It provided crucial insight along the way even though it wound up in the "bad idea" heap in the end.  Thomson, incidentally, won the Nobel Prize in 1906 and Rutherford won it in 1908, both for their work on describing the structure of the atom.  In short, they got it right more than they got it wrong.

Plum pudding, which we know now is not at all like an atom.

Photo by Jules:stonesoup, flickr creative commons

Science, the discipline not the journal, is all about looking for patterns and determining the processes that create them.  An occasional misstep along the way should be expected.  No idea or study should be accepted with full confidence until it is corroborated by other researchers conducting other tests.  In hindsight, it is always possible to say that someone should have known better.  That isn't terribly useful, however. 

As our grasp of science and use of technology grow ever more complex, it is even more important to understand how the process of gaining that knowledge actually works.  The terminus of a glacier is a broken, dangerous mess of calving chunks of ice and stone that continually fall as the ice moves.  The anarchy at the front fails to reflect the cohesion of the massive river of ice that flows inexorably behind.  We should focus as much of our attention on the mass of knowledge behind the advancing front if we are interested in trying to use what we know to our best advantage.

Our survival is going to depend on it.

Johns Hopkins glacier face, Alaska Photo: USGS