TL;DR

• Peer review is a fundamental building block of science
• Most scientists receive no formal on training on how to conduct peer review
• Sharing critical feedback with senior researchers can be intimidating for young scientists
• Journal club meetings are an important way for scientists to learn about new science and how to critically assess science
• Feedback generated in those meetings currently is lost to the world
• PREreview creates a platform for scientists to share and access those meeting notes
• Similarly, peer reviews for journal publication is typically not shared with the world
• Some journals employ open peer review processes, which makes the evolution of a manuscript visible for everyone
• Open peer review also allows for the critical work to become a part of a scientists contribution record. Over time, this could create more incentives for people to conduct peer review
• We need more data on the science of peer reviews, what makes good peer reviews
• Having open peer reviews makes it easier to conduct such research

Daniela and Sam are the cofounders of PREreview, a platform for hosting preprint journal club reviews and discussions.

Follow PREreview on twitter.

Daniela Saderi is a Neuroscience PhD candidate at Oregon Health & Science University, and studies how attention and effort shape behavioral and auditory representation in the brain.

She is also an ambassador for ASAPBio and eLife.

Sam Hindle works as Content Lead at bioRxiv, and before was a post-doc and an assistant professional researcher in Neuroscience at UCSF

She is also an ambassador for ASAPBio, eLife and iBiology

How did you come up with PREreview?

For a while, we have been very frustrated with the current system of publication and have been getting aligned with ASAPbio’s mission, which includes a strong focus on fostering preprints and open peer reviews. Many scientists today aren’t really engaging with preprints. They don’t know how posting preprints or reading them could benefit their careers.

Daniela has been running a journal club for a while. It is a place where you discuss recent papers and highlight positive things as well as flaws and limitations of published papers. But your feedback can’t change what is already published in a peer-reviewed journal, which is very frustrating.

So we started asking ourselves how we can have an impact on things that aren’t published yet as well as have more scientists engaged with it.

That is how PREreview was born.

What is PREreview?

The PRE stand for Post, Read and Engage with preprint reviews.

Capture Journal Club Conversations

So it’s a platform that allows participants of journal clubs who discuss preprints to write up these reviews and post them online. A lot of conversations and critical reviews go on in these journal clubs, but that information is lost to the wider community. So we want to capture that and share it with everybody.

Encourage constructive peer review in early-career researchers

We also want to encourage early-career researchers to be okay about being open with peer review. For that, we provide a number of resources, i.e. on how to do peer review and how to set up and run a journal club.

Encouraging early feedback

PREreview is currently in it’s beta testing phase and is receiving very encouraging feedback from the scientists that have tried it out. Authors whose papers are getting discussed have stated that the quality of the journal club peer reviews is as good as the ones they’ve received from the journals they’ve submitted to.

What is Open Peer Review?

Open Peer Review can actually mean many different things. According to this review, there are at least 22 different definitions of Open Peer Review.

Here’s one way how to categorize the most important ones.

Open Reports

Publishing of peer review reports with the actual paper. People can see what was discussed during the peer review process

Open Identities

Reviewers sign their reports. The community can know who wrote them.

Open Participation

Anyone can contribute, which increases inclusivity and encourages diversity of opinions in the peer review process.

Open Interaction

The idea is topromote interaction between authors and reviewers as well as between reviewers

Which journals are already using an open peer review process?

Recent studies suggest that only about 2% of all journals use some form of open peer review.

BioMedCentral has been doing Open Peer Reviews since 1999.

Other journals and platforms include eLife, f1000R and PeerJ

What happens in a journal club / review meeting?

A group may choose to meet once a week. If you are in charge, you pick a paper you’d like to discuss with the others. You read it and present to the group the main findings.

Then the group discusses the paper and tries to find both strengths and weaknesses of the paper. What is different with our approach is that we a) take a look at preprints instead of peer-reviewed, published articles and b) someone is taking notes and tries to summarize and synthesize what is being said into a feedback that can be shared with the authors and the wider scientific community.

The number of preprints posted online has increased dramatically over the last couple of years. Despite this, the percentage of preprints with associated comments has managed to stay constant at around 10%. However, this is still a small proportion of the available preprints and so we launched PREreview to address this. By bringing preprints into the already established practices of journal clubs, and providing some resources to help early career researchers develop their constructive peer reviewing skills, we hope to provide more feedback to preprint authors. Many people may prefer to give feedback privately but we believe that public commenting has a lot of benefits, as the comments provide a richer understanding for the reader and provide a networking opportunity.

This is what PREreview addresses by generating a collaborative hub for scientists to share their summarized journal club discussions.

Overcoming current challenges in peer review

Sharing critical feedback is intimidating

A survey we ran has indicated that researchers, especially early-career researchers are a bit intimidated to put their comments directly on bioRxiv.

Major reasons for that may be

• Young scientists rarely get formal training on how to conduct good peer reviews
• Sharing critical feedback with more senior researchers is intimidating

So PREreview provides a safe space and a community for them to publish their thoughts. This way they can build confidence in their ability to do peer review.

No formal training in peer review?

Peer review is a fundamental building block of scientific research. As far is we know however, there are very few formal peer review trainings happening in scientists careers.

Several studies have tried to identify if trainings in peer review improve the quality of peer review, and so far there don’t seem to exist any formalized required educational interventions that can reliably improve scientists’ ability to peer review.

One good resource that has just been released is the PLOS reviewer center.

This is very startling. Peer review is what establishes what is good science. Knowing how to conduct good peer reviews should be a fundamental building block of a researcher’s education. Furthermore, developing a “reviewer’s eye” helps trainees to look at their own research with the same scrutiny, helping to make their work more robust.

Benefits of public peer reviews

Additional way to build reputation and get visibility in science

In a closed peer review process, there is no attribution for the peer reviewer. This means that peer reviewers will do the task out of their duty as scientists, but they won’t build their careers with it.

So PREreview also enables researchers to get a DOI on their reviews. If my peer reviews are public and are good, this can become a part of my scientific résumé.

Additionally, just by having my name published next to manuscripts increases the visibility in the field and other scientists may be more eager to collaborate or give priority to my grant applications because they have already heard of me online before.

Better scientists?

We believe that having been mentored in peer reviewing and participating in journal clubs has not only helped usbe better peer reviewers, but also made usbetter scientists. Weare more aware of ourown biases and we now ask better questions. This seems like a strong argument in favor of doing lots of peer reviews and getting feedback on your peer review s.

Better reviews?

Some Randomized controlled studies show that public peer reviews are just as good as than those behind closed doors. Some even suggest they may be better, but the data we have on this is very limited. We definitely need more research into this area.

The ability to conduct studies on what leads to good peer reviews

One critical benefit of having more open peer reviews seems to be the data this will generate. The more peer reviews are in the open, the better we can study the impact of biases, training, relationship between authors etc. to determine what leads to good peer reviews.

What are people’s biggest concerns with open identities?

Fear of retaliation

Young scientists in relatively niche scientific communities are concerned that if they criticize their elder scientists this could result in retaliation when they are applying for grants or wanting their own work to be peer reviewed.

Bio2040’s take

Critical area of research

Peer review is such a critical part of science that any improvement, even a small one could make a huge difference in which results get disseminated amongst researchers, doctors the public and also has huge impacts on funding and careers of scientists.

Sam and Daniela are incredibly passionate about PREreview and open peer reviews and it is hard not to get infected by their enthusiasm. They make some convincing arguments in favor of open peer reviews and putting journal club reviews of preprints online.

As we have seen open peer review currently has no clear definition, as it may include different aspects. The one that is probably most associated with open peer review now is the one of open identity and open reports, ie the peer reviewers putting their names on the reviews and those being shared publicly.

At Bio2040 we love people experimenting with new ways of collaborating and sharing data and opinions online. We definitely like that young scientists have can gain more exposure and confidence with peer reviews. We also like the ability to go back and see how a manuscript has evolved over time with the feedback of the peer reviewers (see an example)

We also think that the data on some of the benefits like improved quality of reviews isn’t that clear yet (Sam and Daniela agree by the way).

However, just by the fact that in order to properly study how peer review is affecting science and how it can be improved, we see a strong case for open peer reviews. The more we can experiment with it, the likelier we are to uncover formulae that are best for all participating stakeholders.

#Onward!

You can find PREreview, Daniela and Sam here

Website: PREreview.org

Email: preprintjc@gmail.com

Twitter: @PREreview_ @Neurosarda @HindleSamantha

About Bio2040

There are so many challenges in drug discovery. We are a group of entrepreneurs and scientists who want to improve things. Our first measure is to educate ourselves and the community on what the biggest bottlenecks and their underlying reasons are. This leads us to discover exciting new opportunities. Bio2040 wants to be the leading place for or entrepreneurs & academics to meet, exchange ideas and launch new ventures.

How to stay connected or get involved

Subscribe to the Podcast on iTunes

Be in the know and sign up for the newsletter.

And you should follow Flavio on Twitter here.