I don't understand our obsession with wanting journal papers to be a source of truth.
I read the Nature News article about the editorial in Science (which you need a subscription to access) announcing that they would conduct a statistical review, in addition to peer review for new papers. While I think reviewing all aspects of a paper carefully is a good thing, I don't think that journal papers need to contain water-tight, fire-proof analyses.
Journal papers should be, and are, exploratory. It is perfectly fine, when you are doing a new experiment to start without a strong experimental design, but just some ideas of what to look for. Once you get your data, it is perfectly fine to 'play around' with various ways of analyzing it until you find something interesting. And it is perfectly fine to publish that. In fact, in my experience, that's what most papers are.
The problem is that we pretend they are something else. Once they are published we pretend that they are a rigorous analyses of an a priori hypothesis and experimental design. We pretend that they are definitely correct. If a paper doesn't stand up to replication we might even retract it so that it doesn't sully the scientific record!
I find this maddening. We need exploratory, open-ended, ill-defined experiments to understand a system, what we are measuring and what limitations we have. You might say that you are supposed to do all that exploration before you design and run the actual experiment you report. To that I say a) how often does that actually happen, b) is it realistic to expect and believe that everyone has done this before publishing their paper and c) why shouldn't you share your preliminary non-rigorous results, with the understanding of what they actually are.
The truth should come later, from replications, meta-analyses and reviews. It should come from the synthesis of many journal papers by different researchers. So instead of wasting time trying to ensure that journal papers are absolutely correct before they are published, why not spend some of those resources building more formalized infrastructure for that second level. Let's stop pretending journal papers are something that they are not.
I read the Nature News article about the editorial in Science (which you need a subscription to access) announcing that they would conduct a statistical review, in addition to peer review for new papers. While I think reviewing all aspects of a paper carefully is a good thing, I don't think that journal papers need to contain water-tight, fire-proof analyses.
Journal papers should be, and are, exploratory. It is perfectly fine, when you are doing a new experiment to start without a strong experimental design, but just some ideas of what to look for. Once you get your data, it is perfectly fine to 'play around' with various ways of analyzing it until you find something interesting. And it is perfectly fine to publish that. In fact, in my experience, that's what most papers are.
The problem is that we pretend they are something else. Once they are published we pretend that they are a rigorous analyses of an a priori hypothesis and experimental design. We pretend that they are definitely correct. If a paper doesn't stand up to replication we might even retract it so that it doesn't sully the scientific record!
I find this maddening. We need exploratory, open-ended, ill-defined experiments to understand a system, what we are measuring and what limitations we have. You might say that you are supposed to do all that exploration before you design and run the actual experiment you report. To that I say a) how often does that actually happen, b) is it realistic to expect and believe that everyone has done this before publishing their paper and c) why shouldn't you share your preliminary non-rigorous results, with the understanding of what they actually are.
The truth should come later, from replications, meta-analyses and reviews. It should come from the synthesis of many journal papers by different researchers. So instead of wasting time trying to ensure that journal papers are absolutely correct before they are published, why not spend some of those resources building more formalized infrastructure for that second level. Let's stop pretending journal papers are something that they are not.
