(Note: I modified this a wee bit on 12.17.06, nothing big, just minor rephrasing.)
In the December 4th, 2006 issue of Chemical & Engineering News is a good article on the issues a researcher may face when their work, or the work of a fellow scientist, cannot be reproduced. The notion of reproducibility is a fundamental pillar on which scientific discovery rests. How can we as scientists ever hope to move forward if the work we rely on for direction and precedence cannot be reproduced?
It should be clear to all my readers that most cases (at the ones I know about) of irreproducible work come the academic world. Industry has multiple checkpoints for a product or a process, so errors or mistakes are caught pretty quickly. Things are different in the Ivory Tower. Let me illustrate this with a story…
I had a postdoc in a large lab (~50 people) that contained researchers from a multitude of fields; from toxicology to organic chemistry. Given that the lab was so large, the PI felt it was reasonable to just give a general research direction to individual researchers. For instance: “Milo, try to develop a better assay for this enzyme.” We would then plan the project, execute experiments and write journal articles on our results, all on our own. More often than not, when we handed a draft manuscript to the PI, it was the first time he was seeing how we approached the project and the data we generated. He was not real good at keeping tabs on his researchers and on their techniques and he seemed lax at challenging our conclusions. I guess he was too busy getting grants.
Now, this system tended to work moderately well (which is quite surprising), most of the researchers were contentious enough to repeat experiments, to ask to opinions from their peers, to use statistics and to generally do all they could to ensure the results were real. However, there was one guy who published two synthetic papers that I needed to follow. I was asked to scale up one of his molecules for a collaborator (the researcher in question had left California at this point), so I pulled up the papers and set to work.
For the life of me, I could not make the molecules. I even had a colleague try the prep, and he failed as well. The boss kept saying: “Milo, it is in the paper, so I don’t see why you cannot follow his prep. It is published!” Now, I am not talking about getting 50% yield instead of the published 79% yield. I am talking about not getting anything. Nada. Zip. Zilch.
So I called the now departed postdoc and explained my problem and asked for advice. Instead of being helpful, he accused me of being a hack and trying to discredit him, this was not the response I was anticipating.
WTF? I just wanted to reproduce his work, but I could not.
Alas, 6 weeks later I had my own route to the molecules. These molecules, as luck would have it, behaved better in the biological model than his “identical” ones.
Was this a case of fraud? I don’t think so. I think that this is a case of poor research ability and poor oversight. Since the PI is the one who will ultimately take credit for positive results, the PI should also move heaven and earth to ensure that their postdocs and student are generating reproducible results. Had the PI in question paid more attention to this particular researcher, he would have known that he was sloppy, did not keep a note book, and was simply a poor chemist.
I don't think a PI needs to check every NMR, every TLC and read the notebooks every night. I also do not expect a PI to give his postdocs and grad student all the answers. I do expect a PI to teach the postdocs and grad students how to do good, solid research, how to think and how to critically analyze data. He also needs to teach them the value of reproducibility. To be blunt, if a PI in unwilling or incapable of doing these things then maybe they need to think of another career path.
