Many of us who have been preprinting our work (and even those who have not) are familiar with the somewhat heightened sense of fear related to posting a preprint relative to submitting a manuscript for peer review at a journal. After all, the world will be able to see our work before reviewers catch issues both large and small. A recent study, showed that work doesn’t change that much from preprint to final version but we can’t shake this overwhelming sense that our papers are improved upon peer review – more polished, more solid, more fleshed out yet more concise. Science sinks in a vacuum and we value feedback, both verbal and written, to make it stronger. Largely we feel that peer review in some form is an absolutely critical aspect to the scientific enterprise. I’m not (at least not in this post) here to argue otherwise. However, given the amount of effort and energy I and many others are putting into trying to bring peer review more broadly to preprints and given this feeling that posting work early comes with this additional fear, I am wondering which aspects and norms of how we prepare our work are actual a byproduct of our current pipeline for peer review. How is the thing we produce different when we plan on a peer review step at journals? And are some of these practices we, as authors, should rethink regardless?
Knowing that a couple of secret people organized by a journal will at some point provide comments on a manuscript:
- Do we limit the amount of feedback we get from other sources prior to journal submission? It takes time, energy, and resources to revise work based on feedback and would we rather spend it on addressing comments that will get us closer to journal acceptance rather than any other feedback that makes the work stronger?
- Do we oversell/over-interpret our results because we are simultaneously trying to make it past reviewers AND editors?
- Is the product that leaves our hands sloppier or more error-ridden if we know we will have opportunities to revise prior to the world seeing it?
- Are we outsourcing our self-worth and the inherent value of our contributions to reviewers by relying on their judgements above our own?
- Are we over-relying on reviewer judgement to decide how we feel about other people’s work?
- Do we limit the amount of critical feedback we provide on other people’s work? Or do we undervalue our time spent on doing this informally when we’re not getting whatever minuscule credit is given for journal-associated review by an editor or our institutions?
- Do we think all feedback on other’s work outside of journal-based peer review needs to be of the same comprehensive type as when only two or three people are responsible for commenting? Or can we more readily provide feedback for egregious errors most relevant to our expertise and move on? And if we do comment in a more modular way without the burden of comprehensive peer review, will there then be more robust commenting from many more sources than exists today?
- Are we generating behemoths that take evermore effort and energy to even read, let alone peer review given journal reviewers will demand and keep moving the goalposts on “completeness” ?
- Do we sufficiently validate the results of prior work before building on it in our own work? And if so, do all papers of all types even need to be peer reviewed?
Sure we can try to replicate the journal system of peer review outside of journals for preprints, but this is a unique opportunity to rethink if there are aspects to the system we can leave behind. In particular, perhaps there is much to be gained for authors and the scientific community if we change the research products we put out and our process to rely less upon the precise nature of the old system in the first place.