Cadmore media player playing video Preprints and the Editorial Work of Journals
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
Preprints and the Editorial Work of Journals
Description:
Preprints and the Editorial Work of Journals
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/5b620b90-b7ae-438c-a8af-f53544604f26/thumbnails/5b620b90-b7ae-438c-a8af-f53544604f26.png
Duration:
T01H09M05S
Embed URL:
https://stream.cadmore.media/player/5b620b90-b7ae-438c-a8af-f53544604f26
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/5b620b90-b7ae-438c-a8af-f53544604f26/AM21 Session 6A - Preprints and the Editorial Work of Journa.mp4?sv=2019-02-02&sr=c&sig=Jsoy9fcXi0cTt4%2BAS2T8GBqtR58ehg%2BNZANpgeo2yV4%3D&st=2024-11-22T09%3A20%3A17Z&se=2024-11-22T11%3A25%3A17Z&sp=r
Upload Date:
2024-02-02T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
JOHN SACK: Welcome, everyone, to the panel session on preprints and the editorial work of journals. Thanks for joining us in today's session. What I'll let you know that's the-- I'm John Sack of Highwire Press, founder. And the meeting hashtag is hashtag SSP2021. You can view this session full screen by clicking on the theater mode button in the video player.
JOHN SACK: Attendees can view closed captions at the bottom of the screen by clicking on the CC icon in the Zoom interface or view the audio transcript to the right of the video by clicking the arrow next to the CC icon and selecting View Full Transcript. For this session, we ask that you place all questions in the chat box to the right of the video player.
JOHN SACK: You won't see chat in the Zoom interface. Your chat has to go into the packable chat box. And I see some people have already put chat in, thanks. That lets us know that we're actually communicating with somebody. The chat on this session will remain active after the session, so you can leave comments or questions and we can continue the conversation. And with that, I will begin.
JOHN SACK: We've often referred to the journal system ecosystem-- and more recently, I've heard the term scholarly publishing ecosystem and scholarly communications ecosystem. All of this reminds us that the wheels of publishing are in contact with lots of other moving parts that are in the hands of authors, readers, students, the media, and even, yes, politicians. And in this past year and a half, especially in biomedical and clinical research, the pressure has been on to discover, communicate, and validate faster than ever.
JOHN SACK: Because millions-- you could even say billions-- of lives were at stake. And when you want to shorten paths in many industries-- that is, when the pressure is on to move faster-- you often do things in parallel by overlapping processes that were serialized previously. And you integrate supply and value chains more than they were in the past.
JOHN SACK: And, of course, not all these integrations and overlaps are successful. So what we want to do with this panel is show the early steps that journals have taken to integrate and overlap their processes with those of preprint services and preprint authors. And we'll talk about what has worked, not worked, may still work, and what ideas there are to come. As I said, if you have questions, put them in the Pathable chat function.
JOHN SACK: We hope to have at least 10, maybe as much as 15 minutes at the end for addressing as many of the questions as possible. So thank you, and now we begin with the brave soul who is the co-founder of two of the premiere preprint services in the life sciences, which HighWire's proud to host, who is also located at an academic institution and by the way, is himself the publisher of several important journals.
JOHN SACK: So over to you, John.
JOHN INGLIS: Well, good morning, everyone. And John, thank you for that typically thoughtful introduction. I am privileged to be here and looking forward to the conversation with my colleagues. So my role really is to set the scene for the subsequent discussions. As John said, preprints are becoming a much more familiar part of the research workflow. I'm going to talk only about the preprint service I know, bioRxiv and medRxiv.
JOHN INGLIS: And I'm talking on behalf of a terrific group of people who work incredibly hard to keep this on the road. bioRxiv and medRxiv are very similar in the sense that they are community-oriented not-for-profits. They're funded by an academic institution called Spring Harbor Laboratory, which has a 100-year history of providing different kinds of services to the research community, and also by the Chan Zuckerberg Initiative.
JOHN INGLIS: These are free services both to submit to and to read-- the content, I should say, is free to submit to and to read. And we see them as services, not publications. They are independent of any publisher or journal, but we work very closely with many publishers and many journals. As is the case with all preprints, they are screened before being posted, but not peer-reviewed.
JOHN INGLIS: And that's an important distinction that I think will come back later in the conversation. And authors are empowered to revise their manuscript any time up to the point where that manuscript is accepted by a journal and gets a DOI. Just some brief data on the two servers. bioRxiv is 7 and 1/2 years old, has a large number of preprints and going up at this point by maybe 3, 3 and 1/2 thousand a month, a large number of authors, a very international distribution of those authors.
JOHN INGLIS: Last year, bioRxiv posted 38,700 preprints. And of that total, 3,000 or so were pandemic-related. And Johnny Coates is going to talk more about pandemic-related preprints later. Substantial amount of usage on both sides, but these numbers from last month are really about half what the maximum usage was in the middle of 2020.
JOHN INGLIS: medRxiv was founded only-- just quite two years ago and has a different set of screening criteria. It serves the health sciences, and we feel that the requirements for posting should be more stringent. I could talk about that in the discussion later. Total number of manuscripts on that archive now is over 20,000.
JOHN INGLIS: Again a very international distribution going up at this point around $1,500 a month. And they-- of-- so the life of medRxiv has overlapped almost completely with the pandemic. And so what has gone on in medRxiv has been dominated by concerns about pandemic-related information. So of all the preprints that have been posted since last January on medRxiv, about 64% of them are directly connected with pandemic-related [AUDIO OUT]..
JOHN INGLIS: Right from the beginning of bioRxiv, we wanted to make sure that preprints could be integrated into editorial workflows. And there are several ways in which that happens. One is we call B2J, bioRxiv to journal. But we have the same process on medRxiv in which an author is given the opportunity to submit her manuscript directly into the submission system of a journal.
JOHN INGLIS: So there are nearly 200 journals available, choices to make on bioRxiv, and just over 100 at this point on medRxiv. Those numbers [AUDIO OUT] up all the time. Another form of integration is the ability an author can be given by a journal to deposit her manuscript on the preprint server at the same time as it's submitted to the journal. That's what we call J2B or J2M.
JOHN INGLIS: There are 37 journals doing that bioRxiv and at the moment, only one on medRxiv, eLife. But we hope-- we are talking to other journals about expanding that possibility. So these are essentially transfer-- manuscript transfer mechanisms, a new form of opportunity for authors we are calling B2X in which an author can send a manuscript to a service, not to a journal, but to a service that does something that will be helpful to the author or to the improvement of the manuscript.
JOHN INGLIS: Often, these are AI-based. And one of those services is going to be turned on in [AUDIO OUT] three weeks, and there will be an announcement. And if anyone in the audience has such a service that they think might fit into this context, then please do get in touch. Most of the preprints on bioRxiv at this point, about 70%, end up with a published version in a journal.
JOHN INGLIS: And we create a link, which you can see here, under the abstract, a link to the journal version. A very small number of journals-- and we would like it to be more-- also provide a link from the published paper back to the preprint version. And there's an example from Plant Direct at the bottom of the screen. We have always been keen to surface the responses to preprints of which there are many different sorts.
JOHN INGLIS: And what we have done recently is to reorganize how we point readers to those evaluations of preprints. So if you look underneath the list of authors there, you'll see a chain of icons that all mean something a little different. The first one is-- shows you the number of comments that we've received, on-site comments on this manuscript.
JOHN INGLIS: The next one shows the number of peer reviews that have been made of this material. And these peer reviews are author opted in. At the moment, they are being conducted by two organizations, eLife and Review Commons. But that opportunity to post peer reviews back to the submitted manuscript, the preprint version of the manuscript, is available to anyone who is conducting that kind of peer review on manuscripts that have a preprint version.
JOHN INGLIS: The next one is a community review. These are typically projects that where a group of individuals or an organization gets together to provide some commentary on a manuscript. This is not author opted in. We provide a link to that kind of content. The automated evaluations I've mentioned, there are not any going at the moment, but we going to start one soon, then links to blogs and news stories, and finally to Twitter.
JOHN INGLIS: And when you click on each of these icons, then they will release this dashboard in which you can see the content in the context of that particular tab. So here are all the tweets. Here are the news stories about this particular paper. No automated evaluations. Here's a link to preLights, which I think Johnny is going to talk more about, and a great example of community review of a preprint.
JOHN INGLIS: And then here are examples of what we call trip transparent review and preprint. And here are some eLife peer reviews which have been posted with the author's consent back to the preprint version. So we are hoping that this is a clearer way of navigating through the different kinds of responses and evaluations that are made to preprints. We've also decided to collect together all the preprints that have peer reviews, author opted in peer reviews.
JOHN INGLIS: And there is a link to that collection on the All Articles page. And then underneath, there are collections of articles by subject category. And you can see off the top there, Animal Behavior and Cognition and Biochemistry. That's just the top of the list of 26 subject categories. And when you go to that collection, you can see-- you can sort by the evaluating entity, as you can see, those MBO-- the Review Commons, eLife, and Peer Richard Science.
JOHN INGLIS: So this area of preprint evaluation is expanding enormously. It has been given a great deal of impetus by the pandemic, and they're setting up of a variety of projects designed to help readers interpret and understand the significance of pandemic-related preprints. And Johnny is going to talk more about the different way in which preprints have been handled by comparison with other kinds of topics.
JOHN INGLIS: So that is my presentation. I look forward to the discussion at the end. And now, my-- I'm going to hand over to Nonia, who is going to take it from here.
NONIA PARIENTE: Let me unmute myself and share my screen. Thank you very much. So-- wait a minute. I'm not sharing the right screen, am I? One second, I have to restart. Share screen, yes. Sorry, where am I? So I am going to give you guys a little vignette about how-- the editorial work plus process with the preprints in this ecosystem of such scholarly publishing.
NONIA PARIENTE: Very briefly, because I assume everybody in this audience will know, but PLOS this is a not-for-profit publisher and an advocacy organization with a mission to accelerate progress in science and medicine. Here is our portfolio, which has very recently grown for the first time in many years in five new journals. So it's the first time I present it complete with PLOS Biology and PLOS Medicine as the flagship journals, and several journals dedicated to publishing sound science with high ethical and technical standards but without considerations of conceptual advance and some community journals in the middle catering to specific communities.
NONIA PARIENTE: So this is an overview of preprint adoption in our published papers in 2020 and 2021. The numbers for PLOS Biology are underestimates because we recently discovered that through a glitch in our system, we actually are missing some that are, for example, being preprinted but submitted to the journal independently of the B2J route that John was speaking of a moment ago.
NONIA PARIENTE: And yeah, and we need to keep in mind when looking at these numbers that, for example, in PLOS One, a lot of the content is out of scope of the most mainstream extreme preprint servers. If we look at our relationship with bioRxiv specifically-- because those were-- The previous table was about papers that had preprint posted anywhere-- these are the papers that either opted in to have PLOS and the positive preprint into bioRxiv, or had already posted a preprint in bioRxiv when arriving at PLOS.
NONIA PARIENTE: And, again, in Bio, there is a slight underestimation of these numbers, I think. And here, you can see a trend over the years. And we're quite happy that in many of the communities, it's clear that our J2B route is actually promoting the deposition of preprints, which, as you'll see at the very end of my presentation, is really one of the organizational objectives of PLOS within the adoption of open science.
NONIA PARIENTE: And so in PLOS neglected tropical diseases and PLOS pathogens, clearly there is an increase in number of authors that have a preprint posted by PLOS after submission of their paper. I'm going to give several vignettes, as I was saying, of different ways in which we've interacted with preprints. We have a really-- we're very big supporters of the preprint movement and are experimenting with different approaches.
NONIA PARIENTE: And so one thing that was not a success story, as John was mentioning in the introduction, was our preprint commenting trial. So for a year, what happened is that comments that were left on preprinted submissions that could be relevant to its scientific evaluation were made available to the handling editors. And these editors-- the idea was that they could use these comments in addition to complementing the peer reviews that they had sourced.
NONIA PARIENTE: We ran this approximately on 1,000 papers across the portfolio for a year only on papers that had opted in to PLOS depositing the preprint for practical reasons of being able to follow the preprints. And what happened was that there were comments, but there were virtually no comments that could inform the peer review process. There were author updates.
NONIA PARIENTE: There were comments saying, this preprint is fantastic or et cetera, but not really critiques of the work or assessments of the work. The ones that were potentially informative often came after peer review had concluded. And this is, again, something I'm going to touch upon in the next slide. It's often that it's coincidental, the posting of the preprint and the submission to a journal.
NONIA PARIENTE: And so if the comment comes substantially after preprint posting then, of course, it won't match with a peer review timing at the journal. And so in the end after one year, we-- PLOS decided that the resources invested in monitoring the comments were not worth the outcome of the trial. Because somebody had to be constantly checking these preprints to see whether there were or not comments.
NONIA PARIENTE: Another thing that we have experimented with is preprint editors. And we've done this in a systematic and in a more ad hoc way. So the systematic way was trialed in PLOS Genetics in 2016. 2016 was relatively early on. And PLOS Genetics wrote an editorial, and they were really pumped up and really excited about the possibility of preprints.
NONIA PARIENTE: And they had a group of associate editors that would look through bioRxiv in their subject area and identify and discuss and invite preprints in a very thorough way. They did this as if these preprints were essentially already at the journal. Several associate editors discussed each one that wanted-- that they were going to call in, et cetera.
NONIA PARIENTE: And I think at the time, the idea was that preprints would be posted not necessarily with a target journal in mind already so that the conversion rates would be relatively high. But what they found and what we now to be the case for a large fraction of authors is that when the preprint was posted, if it had been or it was concomitantly submitted somewhere else. And so they were working and investing time as if the fact of these papers were-- these preprints were submitted to the journal, but then they never came.
NONIA PARIENTE: And some came after a while, but then it was hard to know whether the journal reaching out had an effect on this or they would have come anyway. So the group of-- stopped this official systematic role of doing this across the board a couple of years ago. There are some section editors that do ad hoc preprint calling in, and as do PLOS Biology editors also. And so I have put here for PLOS Biology this ad hoc route is useful for us.
NONIA PARIENTE: I've put here my numbers, so I-- In the last eight months or so, I've called in 50 something, close to 60 preprints with a 61% response rate. So authors do get in touch. Most of them indeed have posted elsewhere. But it's a good strategy for journal outreach nevertheless. And 24% were submitted, which is a very reasonable conversion rate, we think. I do have to note here, though, that I'm a little bit wary-- I mean, I'm happy to do this, and I think it's very useful for the journal from my point of view.
NONIA PARIENTE: But thinking of a future in which perhaps editors would be actively commissioning from preprint servers as the main route of publication, I am really wary of potential biases there. I think that these biases-- the biases everybody talks about in the publication process would likely only increase here. I call in preprints that I see in my social media echo chamber, for example.
NONIA PARIENTE: But we publish a lot of papers that are submitted directly by people having preprints or not, but people that we don't know of work that we are not aware of. And I think yeah, this is a positive thing. This is good. So only direct calling in would make me wary. But anyway, I think it's a good overall initiative. I want to talk a little bit about portable peer review and Review Commons, which is something that John Inglis mentioned, as well as Peer Community In.
NONIA PARIENTE: These are two platforms that PLOS is a member of, and they both provide independent peer review of preprints or of papers that they deposit to a preprint server. And the way this works is they source the reviews in a journal-agnostic way. And then authors can decide to go to any of the journals that are forming part of this platform or this group with those peer reviews.
NONIA PARIENTE: And if it doesn't work at one, then they can try another one. The journals that are members usually reserve the right to add another reviewer if additional expertise is necessary, for example. But with Review Commons, we are seeing more and more that this is less necessary. This is something that started in March 2020. At least we received the first papers in March 2020. And overall, PLOS has received 153 submissions, and we've published 40 papers.
NONIA PARIENTE: Another way that we support the movement, of course, is supporting meta-research on preprints. We support all meta-research and have quite a strong meta-research section. But particularly, we have an increasing number of research on research involving preprints, which is quite interesting. And one of these papers Johnny will talk much more about in a little while, I'm sure.
NONIA PARIENTE: We were asked to explain how COVID has interacted with this whole ecosystem. And for PLOS, there's not really a huge change. Because as I was saying, we were early adopters. And so we already had in place all of these ways of interacting with the preprints. There has been, of course, more volume. More of the preprints that we would have sent to bioRxiv now go to medRxiv, for example.
NONIA PARIENTE: But one thing that was a clear difference is that PLOS signed up to the welcome declaration of research data and finding relevant to COVID. And so what-- beyond just offering the possibility of depositing a preprint to authors whose papers we sent out for review, we strongly encouraged and followed up that they did the positive preprint at the time when a paper was committed to peer review.
NONIA PARIENTE: And we informed the WHO of every submission that went through peer review at PLOS, as was agreed. So I think that was the main kind of operative change that happened. And one last thing I want to talk about is preprints within the PLOS mission. So PLOS has as one of its big four organizational objectives to increase adoption of open science practices throughout the ecosystem, the scholarly communications ecosystem.
NONIA PARIENTE: And within that, increasing the adoption of preprints and increasing trust in preprints is a big part of our effort. We are promoting preprints in our communities as seen by an increased rate of adoption over time. And we're exploring links with other preprint servers. We have very tight links with the Cold Spring Harbor preprint servers that John was talking about. But there is an increasing number, and so we would like to be able to capture the whole preprint ecosystem.
NONIA PARIENTE: And we doing to promote trust in preprints. And one way we're thinking about doing this is surfacing the checks and balances that actually do go into the preprints. One thing I didn't mention but I have thought of is that, for example, in the journal to bioRxiv route, not every paper that wants to be deposited can be deposited. There are preprint fails. And actually, for us, increasingly so.
NONIA PARIENTE: So our preprint checkout rate is around 70% for 2020, down from a higher percentage. Most of the fails have to do with scope issues. But there are checks and balances also in the preprint world, even if they're not peer-reviewed. I think that's the end of my presentation. So if Shalene, you want to go ahead?
SHALENE SINGH-SHEPHERD: Thank you, Nonia. I'm Shalene Singh-Shepherd from Royal Society Publishing. And hopefully, everybody's heard of The Royal Society. But for those who are not familiar with us, we are the UK's independent academy of science based in London. So I am the publishing editor for Proceedings B, which is our flagship biological sciences journal. And we tend to publish content mainly in our core subject areas of ecology, evolution, and behavior.
SHALENE SINGH-SHEPHERD: We were the first channel in our portfolio to appoint a preprint editor in 2017. And that's essentially what I'm going to tell you about today is the whole process we went about in appointing our preprint editor, the processes we use, and also the challenges we've been facing and also some of our successes. Also, another thing to note is that more recently, two of our other journals in the portfolio have also gone down the route of appointing a preprint editor.
SHALENE SINGH-SHEPHERD: And they've also been using similar processes to what we are using. And these two journals are namely Open Biology, which is our online-only fully open access publication that focuses on molecular and cellular biology, and also our sister journal, Biology Letters, which has a similar scope to Proceedings B but publishes short papers.
SHALENE SINGH-SHEPHERD: So we feel that preprint select via archive are a useful resource because they provide a means of providing scientific results to the community very rapidly. And we also feel that from a social justice perspective, they provide a useful means of increasing scholarly productivity, especially for those who may face bias or inequity in the traditional publishing process.
SHALENE SINGH-SHEPHERD: So in 2017, as we noticed preprints were growing in prominence and that more publishers were actually scanning preprints for content, our publishing board took the decision to appoint a preprint editor at Proceedings B. So what we initially did was try to come up with two objectives for the whole initiative to try and see what we would gain from doing such an initiative. And we narrowed it down to we were going to try and raise awareness of the journal across biology, because we are very well known in our three core subject areas of ecology, evolution, and behavior, but not necessarily outside of these areas.
SHALENE SINGH-SHEPHERD: And secondly, what we wanted to do was try to commission more content to try and broaden the scope of Proceedings B, so once again, outside of our core areas. So we were very fortunate that one of our associate editors, Dr. Maureen Nieman from the University of Iowa, volunteered for the role of preprint editor. And bioRxiv was chosen as our server of choice because the subject areas in bioRxiv matched quite neatly with Proceedings B. So what Maureen did was actually turn to a resource in front of her, which was her department of biology at the University of Iowa.
SHALENE SINGH-SHEPHERD: And she actually recruited a preprint team which consisted of graduate students, some of her postdocs, and some undergraduate students. Because she saw this as a perfect opportunity for all of these early-career researchers to interact directly with the scientific process-- sorry, scientific publishing process. So initially, we kicked off in late 2017 with a preprint editorial team headed by Maureen and made up of 20 early-career researchers.
SHALENE SINGH-SHEPHERD: And they came from all areas of biology from the University of Iowa. So this slide just neatly describes the process we are using in Proceedings B. This is obviously evolved over time because from when we started out in 2017 to today, things have obviously changed and we've learned from our mistakes and tried to improve on the process.
SHALENE SINGH-SHEPHERD: So essentially, each of the team members in the preprint team is assigned a particular subject area. And they tend to scan papers in bioRxiv on a monthly basis. Promising papers are then assessed more thoroughly, so they look at the entire paper and not just the abstract. And papers are then selected and added to a database, which the team then meets and evaluates. And Maureen will then make a final assessment as to which of these papers she'd commission to invite for the journal.
SHALENE SINGH-SHEPHERD: We then send out a personalized email to the authors of the paper. And if they choose to accept our invitation, the paper then goes through peer review and hopefully gets published in Proceedings B. How the process has developed over time is that more recently, we've appointed two associate editors within the team. They are still from Maureen's preprint team, but their main role is to decide which of the preprint members would be assigned to which subject categories to handle communications for the team.
SHALENE SINGH-SHEPHERD: And they also handle recruitment. So how have things been going in Proceedings B? The initiative has been running since 2017, and thus far we've had a lot of positive feedback from authors. A lot of them have come back and said to us that they may not have considered Proceedings B as their journal of choice in the past. And they may do so in the future, or they may even come to us if the paper's rejected elsewhere.
SHALENE SINGH-SHEPHERD: But two main things as we have noticed about papers that are already on bioRxiv is that one, they all have a very high standard. But secondly, they have also been submitted to another journal before we approach them. So I think this is just something that other people have seen as well. We have found though that we have been successful in our outreach.
SHALENE SINGH-SHEPHERD: So one of our objectives was to raise awareness of Proceedings B, especially in areas outside of our core. And we have found that this initiative has been very helpful in trying to attract authors who may not have considered us in the past and encouraging them to submit to Proceedings B. In my slide, I've put up some headline stats for you. We've recently evaluated all of the data we have from our preprints initiative.
SHALENE SINGH-SHEPHERD: And this has been analyzed and written up into a research paper, which is currently at review. But what I can share with you today is that to date, we have sent out at least 1,200 invitation emails to authors of bioRxiv papers. From that, we've had over 100 papers submitted to the journal. And we've had about 30% of these papers accepted for publication.
SHALENE SINGH-SHEPHERD: The acceptance rate for the solicitor's papers doesn't differ very much from the acceptance rate of papers traditionally submitted to Proceedings B. But what we have found is that in certain subject areas, these solicited papers have been higher. So if I use the example of two subjects of synthetic biology and bioengineering where Proceedings B traditionally sees a very low number of submissions, the number of papers that have come via the preprint route is far higher.
SHALENE SINGH-SHEPHERD: One other thing that we have done was also to do a survey of the preprint editorial team members, just to get their feedback on how they feel the initiative has been going and also their thoughts on preprints. And what we found that many of the early-career researchers involved have said that this was a great opportunity for them to learn more about the publishing process and to engage with it, and also to learn more about the things like the scope of a journal and what peer review actually entails.
SHALENE SINGH-SHEPHERD: Many have also said to us that they found that their academic career was improved, and it aided their awareness and knowledge of their research area. And for preprints as a whole, they found it a useful resource, as it made science and scientific results available to both scientists and non-scientists.
SHALENE SINGH-SHEPHERD: But I have to say, it hasn't been smooth sailing throughout the whole four years that we've been running our preprints initiative. One of the main problems we found initially was that some authors thought that our invitation email from the preprint editor was spam and from a predatory journal. So we've had to work hard to actually personalize our email just to make it very clear that this is actually from a genuine journal.
SHALENE SINGH-SHEPHERD: We've also had this very awkward issue of we've sometimes invited authors from bioRxiv papers, but it turned out that these authors had already submitted to Proceedings B where they were rejected. So, I mean, that is a very difficult situation to be placed in. So what we do now is make sure we cross-reference all of our invitations against a list of rejected papers already from the journal to ensure that we don't do this again.
SHALENE SINGH-SHEPHERD: Another issue we've faced is journal limitations. So for instance, at Proceedings B, we have a 10-page limit for our articles. And that can be very tricky when we've invited authors from a bioRxiv paper which is far longer, and then we're asking them to reduce the length. Sometimes they're not happy with us asking that. And we've also had to manage author expectations. So essentially, we've had to make it very clear in our invitation letter that we're inviting you to submit the paper.
SHALENE SINGH-SHEPHERD: We'll still go through traditional peer review. And this isn't a guarantee of publication, because we've had a few issues of that misconception in the past. One final thing that we have found is that this whole initiative is trying to broaden the scope of Proceedings B. So we're trying to solicit papers outside of our core areas.
SHALENE SINGH-SHEPHERD: But at the same time, some of these papers that we have invited have been rejected before review by our editorial board members handling them for being out of scope. So this problem is kind of circular in nature. It's very difficult to try and broaden the scope when you don't have the right expertise on the editorial board. So after much discussion, what we decided to do was to introduce a new article type within Proceedings B to accommodate a vast number of papers that our preprint editor had identified that were focused on scientific practices.
SHALENE SINGH-SHEPHERD: In this way, we could then expand our scope to cover that area as well as having more submissions coming in. So what we did in 29 was launch biological science practices as an article type. We have a new associate editor to handle just these submissions. And so far, it's been proving quite popular with, I'd have to say, about 40 to 50 papers already submitted.
SHALENE SINGH-SHEPHERD: And I think to date, we have about seven published. And just finally, I'm going to end on some feedback on what the pandemic has impacted at Proceedings B. So we have seen an unprecedented number of submissions to the journal. I mean, similarly to an increase in papers submitted to bioRxiv, we've also subsequently had more papers being submitted to the journal by this route. We've taken on board the fact that a lot of authors would like a quick transfer of papers, and we've now signed up to the direct transfer from the journal-- sorry, from bioRxiv to the journal.
SHALENE SINGH-SHEPHERD: Especially with COVID papers, we found that authors were very eager to get their publications out as soon as they could, and having a direct transfer has sped up the process a lot more. Our preprint editor also decided to make the preprint editorial team global. So what we did in 2020 was a recruitment drive to try and encourage early career researchers from around the world to join our preprint team.
SHALENE SINGH-SHEPHERD: And to date, we now have 60 members of the team from 29 different institutions and 10 countries across the world. And we're finding this a lot more useful because we have a wider breadth of expertise to now evaluate a wider breadth of submissions. And just one final thing I mentioned at the beginning is that we've expanded now the preprint initiative to other Royal Society journals.
SHALENE SINGH-SHEPHERD: And two of them now, Biology Letters and Open Biology, have also appointed preprint editors. So at this point, I'm going to end there and hand over to our next speaker, Johnny.
JONATHON COATES: OK, thank you. So this story here is actually a really good example of how preprints are beneficial for early-career researchers. I'm a postdoc. I'm a senior also. Hopefully, there's going to be two papers out of this. And I think this is just a really nice example of why all early-career researchers should preprint. So I'm going to put some numbers on a lot of what's being talked about so far, and how the pandemic has affected preprints in particular.
JONATHON COATES: So we know that the pandemic led to an increase in papers. And so we looked at the first 10 months, so from January to October, November time. And in that time period, you can see there's just under 125,000 papers, preprints published in that time frame that are on COVID. And of that, about 25% were preprints, which is a huge number. Normally, preprints represent about 3% of the biomedical literature, which interestingly last year, that went up to 8%.
JONATHON COATES: So there's been a huge impact on preprint usage purely because the pandemic. And what I particularly liked in some of them-- I tried to distill this down to a very small number of slides. But what you're seeing here are on your left, you've got your COVID preprints. And on your right, you've got your non-COVID preprints. The bars that are filled in are those where the senior author has previously posted a preprint.
JONATHON COATES: The bars are not shaded in, or the lighter bars, are those where authors have not previously posted a preprint. So those represent people who are coming to the preprint service for the first time. And what you're seeing across all of the countries we looked at is a significant portion of people turning to preprint service, especially for COVID research, for the first time, which suggests that there's been a radical rethink in how people are approaching sharing their work.
JONATHON COATES: Mainly, I suspect that's driven by the fact that we need to get things out quickly. But there may be other reasons for that. And if we look at-- so this is just from bioRxiv and medRxiv. If you look at our preprint, you'll see that we've looked at other service as well. But bioRxiv and medRxiv are really good examples of preprint servers because they have APIs that you get access really easily to a lot of this underlying data.
JONATHON COATES: So we look at those two servers. What you're seeing here are the abstract views and the PDF downloads for COVID research, which is in red, or in green, the non-COVID preprints. And basically, what we found within that sort of 10-month period is that the COVID preprints are being viewed 18 times and downloaded almost 30 times more than non-COVID preprints were, which is an incredible demand. And that kind of led us to suggest that maybe this is not all coming from scientists, which makes sense if you keep up with the news.
JONATHON COATES: So we looked at where these preprints were being shared. So this panel on the top left here is the percentage of preprints that have more than one citation. And in red, you're seeing COVID. And in green, you're seeing non-COVID. And you can see the stark difference, especially for those preprints that were posted early in the pandemic, so within the first four months.
JONATHON COATES: And although that has decreased, there's still a significant difference at the end of October. This decrease is not just fatigue or anything like that. This is most likely due to our collection window. So we collect this data on the first of November. So those posted in October haven't had time to accumulate citations. We also looked at Twitter. A lot of preprints are tweeted out by bots.
JONATHON COATES: And there's no easy way to filter that out. So what we did was we took all of the preprints that had more than two tweets each. And you're seeing the percentage of that here. So, again, the top line, almost 100% for the entirety of the year is the COVID preprints. And this slightly lower line is the non-COVID preprints. So COVID preprints are being heavily shared across Twitter. And that is not just by scientists, which I'll get to.
JONATHON COATES: We also looked at how news organizations have been using preprints. So you'll see-- and if you look before January actually, you'll see the preprints that are not COVID-- so the bottom line-- generally are not reported on in the news. And I've talked to a number of journalists in doing this work. And generally, the line was journalists do not report preprints.
JONATHON COATES: And you can see that very clearly changed when the pandemic hit, but particularly when the pandemic was kind of really starting to take off. And what you're seeing is that on average, COVID preprints, about 25% of the COVID preprints are reported on in at least one news article, often more than one. And throughout that time period, almost no preprints that are not covered are reported on by news organizations.
JONATHON COATES: The other thing we looked at were whether or not preprints were driving policy changes. I had a strong suspicion early on that they were, but it would be nice to have some data for this. So we looked at the European Center for Disease Control, the World Health Organization scientific briefs, and the UK post based in the UK parliament. They have this service which they write little briefs and they are then used to help inform MPs making decisions and policy-making decisions.
JONATHON COATES: And across those three things that we looked at, you can see COVID preprints are being referenced, not in a huge amount. Sort of roughly 20% of the references in each article were a preprint. But the most interesting part of this data is that when we looked at non-COVID preprints in this time frame, so January to October, all before that, we couldn't find any preprints used in policy documents.
JONATHON COATES: So you're seeing quite a significant shift in how everybody seems to be using preprints for the pandemic. And unfortunately, not all this use is good. So we looked at the top 10 preprinted papers, so COVID and not COVID, and how they were received on Twitter. So what you're seeing here is an example of a preprint that was posted, shared on Twitter, quite widely shared on Twitter.
JONATHON COATES: And I'm just pointing out some of the hashtags we noticed in here. And you see things that are kind of representative of things you would want to be getting across, so disinfect, airborne, social distancing, preprint signs are in there. But what you're seeing are things that the preprint is talking about, and those are getting fairly well delivered across Twitter, at least.
JONATHON COATES: So the scientific message is getting out. On the other hand-- and unfortunately, at the time of this data collection was the second highest tweeted about article-- some preprints are being used by right-wing conspiracy groups, right-wing politicians largely, and a lot of people who are quite racist, unfortunately. And what you see here is that this preprint was, I've said, hijacked.
JONATHON COATES: So you can see hashtags to do with COVID-19 being a lie, fake news, pandemic, ending lockdowns. Bill Gates is in there. Dr. Fauci gets mentioned a lot in these kind of hashtag word clouds. And you're seeing a very clear dichotomy between science getting shared properly and just complete misuse, misunderstanding of what the message should be.
JONATHON COATES: So we really need to think a bit more about how we are actually sharing our work and how we police this. This could have a whole paper on its own. We did not have time to go beyond this, unfortunately. But this really importantly leads to the question of are preprints good enough quality to share? And it's something people have been asking for quite a long time.
JONATHON COATES: And there's been in the past couple of years a few papers that have kind of looked at that topic, but not quite. So we decided to directly address the question of quality between preprint and its published version. We only looked between January and April because this allowed us to look at all of the preprints that had been published that were COVID related, and then also get a date set for a control.
JONATHON COATES: So the first thing we did was we looked at the total number of panels and tables in a preprint or its paper companion. And we just count those up to see if there was a significant amount of extra work being put in there. You can see that the COVID on the left here are lower in terms of the panels and tables than non-COVID, which makes sense because generally, those were also shorter papers.
JONATHON COATES: This is emerging, so you'd expect those to be a bit shorter. But you can see that whether it's COVID or not COVID, between the preprint and the paper, on average there's no difference in the total number of tables and piles. And when we categorized how these were changing where they did change-- so I said over 60% didn't have any changes-- a large portion just had rearrangement, so a figure might be moved from a panel-- from a main figure might be put into a supplementary, for example, or something was moved.
JONATHON COATES: We did notice that roughly 20% had significant content added. So this suggests that there is a very important place for peer review, and that peer review is working and having a positive impact on papers as they progress. And then we also looked at abstracts. And the reason we focused a lot of our analysis on abstract is because we know that a lot of the general public are using preprints and papers. And normally, outside of COVID, a lot of these things are behind paywalls if it's a paper.
JONATHON COATES: And so often, the only thing people will read is the abstract. And the abstract generally distills the key messages of paper. So we reasoned that if the abstract changes, then there's been a key change in one of the conclusions of the paper. We tried a bunch of different automated ways of doing this to try and get some kind of ideas whether the preprint abstract was changing when it was published or not.
JONATHON COATES: So the higher the change ratio for both of these, the more a preprint paper abstract changed. Generally, we didn't see a huge difference, so they were roughly the same. So COVID preprints generally had a slightly higher change in their abstract than non-COVID articles did when they were published. But because these automated methods didn't really line up very well, we decided to do a manual analysis, which I must say, it took a very long time, and I will happily never read another abstract again.
JONATHON COATES: But when we did this, what we found was that for 6% of non-COVID research or 15% of COVID research, the abstracts undergo a discrete change in their key conclusions. This is really important. But if we reframe that, 94% of the non-COVID literature or 85% of COVID literature that were preprints that were then published had no discrete changes in their conclusions, at least within the abstracts.
JONATHON COATES: So this suggests that a lot of the anecdotal evidence you might have with that preprints are generally high quality. This really suggests that they are comparable to the preprint of the published literature. And not that we should treat preprints with the same level that we might treat with the published literature, but that we should treat published literature the same as preprints.
JONATHON COATES: And that is, we should approach both with skepticism and caution. We should read those papers and analyze them ourselves, not just rely on peer review. So we found that the rates of preprint posting were 100 times that of previous epidemics, represent the core of all articles. Authors from all over the world, primarily UK, US, and China-- they were getting viewed and downloaded considerably more than non-COVID preprints were.
JONATHON COATES: And the quality was not detectably different to non-COVID preprints. And we surmise basically that there's been a cultural shift in how preprints are being used that we hope is here to stay. And to end on a nice little positive, a lot of the co-authors on this paper I actually found through Twitter or through just kind of approach that I thought would be really good to get involved with.
JONATHON COATES: And all the authors at the time were early career researchers, which is also I think is a-- just a great thing for a paper. So you can read a lot more about this in our PLOS Biology paper. If you're interested in the changes analysis that we did, that-- we actually submitted that to PLOS Biology with another group from the US fairly recently. But that's currently on bioRxiv. And I have more slides available, a wider slide deck if you want those.
JONATHON COATES: Thank you. Back to you, John.
JOHN SACK: Thanks very much. OK, good. We have-- let's see, 1, 2, 3, 4, 5-- I think six or seven questions to the panelists. So I will try and take us through those. SSP has told us we can go a few minutes over. And we'll use that to try and address all the questions. Let me get back to the first one-- let's see, from Kimberly.
JOHN SACK: Kimberly asks-- and I think this is a-- probably a question for any number of folks. But I think John is probably-- John Inglis has probably handled it before. How is it that an author can select an OA license for a preprint and then assign it-- assign the copyright-- to a publisher for the published version?
JOHN SACK: John? Unmute, John. There we go.
JOHN INGLIS: So if I understand the question, the questioner is saying, is it OK to choose any kind of license on a preprint without regard to the requirements at the journal? And as far as we are concerned, the author owns the copyright on every preprint and chooses the license. And my understanding is that most journals have no problem with transition of a paper from a preprint version to a journal-published version with a different kind of license.
JOHN INGLIS: Shalene is nodding her head, and so thank you for confirming that.
JOHN SACK: There's a related question that I think is pretty quick to answer from Cami. Does a preprint get a DOI, or is a DOI assigned only to the version of record? Why don't we have John take that one as well?
JOHN INGLIS: Every preprint has a DOI, and it is specific to the server. And, again, the convention has arisen that the published version will have the DOI of the journal concerned.
JOHN SACK: So everybody gets a DOI. The next question came from Anne Stone. Hello, Anne. It's for Shalene. Regarding the opportunity to create more equity, has the data shown any impact?
SHALENE SINGH-SHEPHERD: We've analyzed that data. The only difficulty we've had is that' we don't always have information, that more information that we need on authors selected in the submission system that we're using. But that's something we're currently working on at the Royal Society to try and improve on our system before we could give a definitive answer of how much of an impact we've had using this initiative.
JOHN SACK: John Long asks a question I think is also intended for Shalene. How have you gauged enhanced awareness of the journal?
SHALENE SINGH-SHEPHERD: I think at the moment, we're purely doing that on the response rate that we're getting from authors that we solicit. We are collecting information from everyone who actually has come back and responded to our emails. So in the fact that we know we've sent out 1,200 emails, we've also collected data on the number of people who have actually responded. And in that way, we've been measuring the kind of response that we've been getting.
SHALENE SINGH-SHEPHERD: But, of course, not all of the responses we've had have been positive. We have had a few people come back very negatively or who have told us never to approach them again. But we're currently basing data that we collect in that way on how we are measuring whether this outreach has improved for proceedings or not.
JOHN SACK: So I think the next question from George could go to Shalene and Nonia both, if you both commented on this, in talking about preprint editors. The sense of the main reason, do you have a sense of the main reason authors choose not to submit to the journal if they were sent a solicitation email? Is it because it's spam? Is it because it was already submitted somewhere else, or they didn't want to publish the paper in its current form?
JOHN SACK: Nonia, why don't you take that first and then Shalene.
NONIA PARIENTE: So for us, clearly it's because it's somewhere else. I say 70% to 75% of people we approach say the paper is somewhere else, or they are just about to submit it somewhere else. So they deposited a preprint with a very clear, specific place of submission in mind. Having said that, we use portable peer review, for example. So we encourage transfers from other journals with the peer-reviewed source that the other journals-- so we have seen some of those papers eventually make it to us.
NONIA PARIENTE: The fact that it's submitted somewhere else doesn't mean it's going to be successful where it's submitted. But I would say that there are two very clear demographics also for my interactions with authors. There are the people who are very community-build approach. And so they really preprint a paper to get community feedback and improve the paper. Most of those feedback is usually in an email or in a personal communication, but it doesn't mean it doesn't exist.
NONIA PARIENTE: It does. And so there are, I would say, perhaps 25% to 30% of people who actually do it for that. And those people will-- are very open minded about where to ultimately submit. And the others do it almost concomitantly.
SHALENE SINGH-SHEPHERD: I mean, I would have to agree with that and say it's been similar for us that people have already submitted their papers elsewhere by the time our preprint editors approached them. They've come back and said, well, if it's been rejected elsewhere, we may reconsider and submit to you, which for us is very helpful. But I think that's one of the main issues we've run into, as well as the fact that sometimes people interpret our invitation as spam from a predatory journal.
SHALENE SINGH-SHEPHERD: But I think because we've now personalized what we say in those invitations, that has helped us somewhat. But I think yeah, the main problem is still when people post on bioRxiv, the paper is also simultaneously submitted to a journal.
JOHN INGLIS: John, maybe I could just say a couple of years ago, we did a survey of about 4 and 1/2 thousand authors. And we found in our survey that 40% of them said that they simultaneously deposited a manuscript on bioRxiv and submitted to a journal. But 40% of them said they just submitted to the preprint server. So one of the things that we have been thinking about a lot recently is how to allow authors to signal their interest in being approached by editors.
JOHN INGLIS: And that would seem like a useful service for authors. But on the other hand, we are also hearing from authors who get an enormous amount of solicitation from journals that they do not want to hear from. So balancing those two concepts is tricky, and we are still thinking that through.
JOHN SACK: Thanks, John. I think the last question in the list or next to the last question is for Johnny. Have you gone back to the journalist to find out the implications of preprints in reporting, and will they go back to not reporting preprints?
JONATHON COATES: It will be a small sample size. But generally, they seem to quite like preprints. There's no embargo on them. One journalist actually pointed out that he really likes the bioRxiv website because it's laid out so easy for you to pick out what you want and what category you want. So from the people I've spoken to, they will probably be sticking with it.
JOHN INGLIS: Again, John, could I jump in there? I've been talking with journalists too like Johnny. And one thing we have to bear in mind is that when the pandemic hit, most-- even science journalists were unfamiliar with preprints, because they were-- They had been a thing in physics and math. And let's face it. Most science journalists do not report on physics and math. So in the beginning of 2020, there was an enormous amount of confusion about how to handle preprints.
JOHN INGLIS: And I think talking to folks that-- say, The New York Times and other places, they now have really got their methodology down. And they know both of the pluses-- as Johnny said, the pluses are instant access to the latest research-- but also how to assess how other experts think about this particular work. And many journalists have their own sort of little brains trust of people, trusted authorities who they can go to for opinions.
JOHN INGLIS: So I think we're seeing an enormous improvement in how parents are reported, and also an increase in the frequency with which they are noted as not being peer reviewed, which was another thing that didn't happen as much as it should have done in the early days of the pandemic.
JOHN SACK: And a couple more questions just came in. Maybe they're coming in faster than we're answering them. Let's see. Adam has a question for Shalene. Can you share the percent of solicited papers which were ultimately accepted? I think I saw that number.
SHALENE SINGH-SHEPHERD: Up to date, we've had about 30% that were from the preprint streak that we've accepted and published.
JOHN SACK: That's significant. The question is from Sarah.
NONIA PARIENTE: From the ones that were submitted, but I guess the question is from the solicitation, so from the 1,200.
SHALENE SINGH-SHEPHERD: Yeah, we had about over 100 submitted. I think it was about 100 and--
JOHN SACK: Yeah, that's right, 100 and-- yeah, it was about 1 in 120.
SHALENE SINGH-SHEPHERD: Of the 30% were accepted.
JOHN SACK: Question from Sarah. Why don't we send this one to John, John Inglis? Question about where preprint is revised, how are revisions typically tracked? And is a record of the exact revision made available to readers?
JOHN INGLIS: All the revised versions live on the preprint server, so-- and can be looked at separately. We do have a tool for allowing authors to choose to note the changes that have taken place from version to version. It's not something we-- a relatively small proportion of authors actually do that.
JOHN SACK: And final toss-up question for all the panelists who care to comment-- what's the fake news out there about preprints? Are there any misinformation kinds of things that you'd like to correct, especially publishers' perceptions of? Why don't we start John, Nonia, Shalene, and then Johnny.
JOHN INGLIS: Gosh, where to start. That's a-- [LAUGHTER]
JOHN SACK: Choose just one, John. [LAUGHTER]
JOHN INGLIS: OK, well, I think the one I would choose is that this is somehow-- that preprints are second class research outputs. I think the data that Johnny showed and other studies of their comparisons between preprint versions and peer-reviewed versions show that when the adjustments take place, they can be significant. But there is not-- I think the events of the last few years have demonstrated that a preprint version is not some inferior output that can be ignored or should be ignored.
JOHN INGLIS: And I think that that was one perception that we encountered quite a lot at the beginning of bioRxiv.
JOHN SACK: Nonia?
NONIA PARIENTE: I think one of the bigger things I have to dispel all the time is that anything goes, that it's like a dumping ground for anything. It stems from lack of appreciation and knowledge, but also from the fact that, for example, some don't have methods or what have you. And so I really stress the checks and balances. As John was mentioning at the beginning, there are screens. It's not peer review, but it's-- they are screened.
NONIA PARIENTE: And indeed, from having the experience of the J2B workflow, we know that not everything will fit. And it's not that just anything can be there. I wanted to make a very little comment, if I may, on these comparisons of preprints and published versions. And this is semi-anecdotal but true, and I've worked in many publishers over my career. Authors do not revise their abstracts, no matter how much they've changed their paper.
NONIA PARIENTE: This just happens. They throw the title in an abstract two years ago, and that's what gets to pre-publication unless an editor steps in. But in response to Johnny's preprint, I've actually changed our revision letters to stress that fact. But it really depends on how detailed the editorial work is whether or not you would change-- you would see change in an abstract, even if changes have occurred.
JOHN SACK: So I think we're going to get cut off in less than a minute automatically. Shalene, anything on this topic?
SHALENE SINGH-SHEPHERD: I mean, I would just emphasize quality, that quality of preprints is exceptional. And the misconception that, as Nonia said, anything goes, that's clearly not true from what we've been finding.
JOHN SACK: Johnny, last word?
JONATHON COATES: Same thing, that preprints are just as good as papers for the most part, and they're probably the greatest thing at leveling the field a bit for early-career researchers. I could not have done this without that.
JOHN SACK: Thank you all very much. We appreciate the attention of so many attendees and also excellent panelists. Thanks.
JOHN INGLIS: Thank you all.
NONIA PARIENTE: Thank you.
Segment:0 .
JOHN SACK: Welcome, everyone, to the panel session on preprints and the editorial work of journals. Thanks for joining us in today's session. What I'll let you know that's the-- I'm John Sack of Highwire Press, founder. And the meeting hashtag is hashtag SSP2021. You can view this session full screen by clicking on the theater mode button in the video player.
JOHN SACK: Attendees can view closed captions at the bottom of the screen by clicking on the CC icon in the Zoom interface or view the audio transcript to the right of the video by clicking the arrow next to the CC icon and selecting View Full Transcript. For this session, we ask that you place all questions in the chat box to the right of the video player.
JOHN SACK: You won't see chat in the Zoom interface. Your chat has to go into the packable chat box. And I see some people have already put chat in, thanks. That lets us know that we're actually communicating with somebody. The chat on this session will remain active after the session, so you can leave comments or questions and we can continue the conversation. And with that, I will begin.
JOHN SACK: We've often referred to the journal system ecosystem-- and more recently, I've heard the term scholarly publishing ecosystem and scholarly communications ecosystem. All of this reminds us that the wheels of publishing are in contact with lots of other moving parts that are in the hands of authors, readers, students, the media, and even, yes, politicians. And in this past year and a half, especially in biomedical and clinical research, the pressure has been on to discover, communicate, and validate faster than ever.
JOHN SACK: Because millions-- you could even say billions-- of lives were at stake. And when you want to shorten paths in many industries-- that is, when the pressure is on to move faster-- you often do things in parallel by overlapping processes that were serialized previously. And you integrate supply and value chains more than they were in the past.
JOHN SACK: And, of course, not all these integrations and overlaps are successful. So what we want to do with this panel is show the early steps that journals have taken to integrate and overlap their processes with those of preprint services and preprint authors. And we'll talk about what has worked, not worked, may still work, and what ideas there are to come. As I said, if you have questions, put them in the Pathable chat function.
JOHN SACK: We hope to have at least 10, maybe as much as 15 minutes at the end for addressing as many of the questions as possible. So thank you, and now we begin with the brave soul who is the co-founder of two of the premiere preprint services in the life sciences, which HighWire's proud to host, who is also located at an academic institution and by the way, is himself the publisher of several important journals.
JOHN SACK: So over to you, John.
JOHN INGLIS: Well, good morning, everyone. And John, thank you for that typically thoughtful introduction. I am privileged to be here and looking forward to the conversation with my colleagues. So my role really is to set the scene for the subsequent discussions. As John said, preprints are becoming a much more familiar part of the research workflow. I'm going to talk only about the preprint service I know, bioRxiv and medRxiv.
JOHN INGLIS: And I'm talking on behalf of a terrific group of people who work incredibly hard to keep this on the road. bioRxiv and medRxiv are very similar in the sense that they are community-oriented not-for-profits. They're funded by an academic institution called Spring Harbor Laboratory, which has a 100-year history of providing different kinds of services to the research community, and also by the Chan Zuckerberg Initiative.
JOHN INGLIS: These are free services both to submit to and to read-- the content, I should say, is free to submit to and to read. And we see them as services, not publications. They are independent of any publisher or journal, but we work very closely with many publishers and many journals. As is the case with all preprints, they are screened before being posted, but not peer-reviewed.
JOHN INGLIS: And that's an important distinction that I think will come back later in the conversation. And authors are empowered to revise their manuscript any time up to the point where that manuscript is accepted by a journal and gets a DOI. Just some brief data on the two servers. bioRxiv is 7 and 1/2 years old, has a large number of preprints and going up at this point by maybe 3, 3 and 1/2 thousand a month, a large number of authors, a very international distribution of those authors.
JOHN INGLIS: Last year, bioRxiv posted 38,700 preprints. And of that total, 3,000 or so were pandemic-related. And Johnny Coates is going to talk more about pandemic-related preprints later. Substantial amount of usage on both sides, but these numbers from last month are really about half what the maximum usage was in the middle of 2020.
JOHN INGLIS: medRxiv was founded only-- just quite two years ago and has a different set of screening criteria. It serves the health sciences, and we feel that the requirements for posting should be more stringent. I could talk about that in the discussion later. Total number of manuscripts on that archive now is over 20,000.
JOHN INGLIS: Again a very international distribution going up at this point around $1,500 a month. And they-- of-- so the life of medRxiv has overlapped almost completely with the pandemic. And so what has gone on in medRxiv has been dominated by concerns about pandemic-related information. So of all the preprints that have been posted since last January on medRxiv, about 64% of them are directly connected with pandemic-related [AUDIO OUT]..
JOHN INGLIS: Right from the beginning of bioRxiv, we wanted to make sure that preprints could be integrated into editorial workflows. And there are several ways in which that happens. One is we call B2J, bioRxiv to journal. But we have the same process on medRxiv in which an author is given the opportunity to submit her manuscript directly into the submission system of a journal.
JOHN INGLIS: So there are nearly 200 journals available, choices to make on bioRxiv, and just over 100 at this point on medRxiv. Those numbers [AUDIO OUT] up all the time. Another form of integration is the ability an author can be given by a journal to deposit her manuscript on the preprint server at the same time as it's submitted to the journal. That's what we call J2B or J2M.
JOHN INGLIS: There are 37 journals doing that bioRxiv and at the moment, only one on medRxiv, eLife. But we hope-- we are talking to other journals about expanding that possibility. So these are essentially transfer-- manuscript transfer mechanisms, a new form of opportunity for authors we are calling B2X in which an author can send a manuscript to a service, not to a journal, but to a service that does something that will be helpful to the author or to the improvement of the manuscript.
JOHN INGLIS: Often, these are AI-based. And one of those services is going to be turned on in [AUDIO OUT] three weeks, and there will be an announcement. And if anyone in the audience has such a service that they think might fit into this context, then please do get in touch. Most of the preprints on bioRxiv at this point, about 70%, end up with a published version in a journal.
JOHN INGLIS: And we create a link, which you can see here, under the abstract, a link to the journal version. A very small number of journals-- and we would like it to be more-- also provide a link from the published paper back to the preprint version. And there's an example from Plant Direct at the bottom of the screen. We have always been keen to surface the responses to preprints of which there are many different sorts.
JOHN INGLIS: And what we have done recently is to reorganize how we point readers to those evaluations of preprints. So if you look underneath the list of authors there, you'll see a chain of icons that all mean something a little different. The first one is-- shows you the number of comments that we've received, on-site comments on this manuscript.
JOHN INGLIS: The next one shows the number of peer reviews that have been made of this material. And these peer reviews are author opted in. At the moment, they are being conducted by two organizations, eLife and Review Commons. But that opportunity to post peer reviews back to the submitted manuscript, the preprint version of the manuscript, is available to anyone who is conducting that kind of peer review on manuscripts that have a preprint version.
JOHN INGLIS: The next one is a community review. These are typically projects that where a group of individuals or an organization gets together to provide some commentary on a manuscript. This is not author opted in. We provide a link to that kind of content. The automated evaluations I've mentioned, there are not any going at the moment, but we going to start one soon, then links to blogs and news stories, and finally to Twitter.
JOHN INGLIS: And when you click on each of these icons, then they will release this dashboard in which you can see the content in the context of that particular tab. So here are all the tweets. Here are the news stories about this particular paper. No automated evaluations. Here's a link to preLights, which I think Johnny is going to talk more about, and a great example of community review of a preprint.
JOHN INGLIS: And then here are examples of what we call trip transparent review and preprint. And here are some eLife peer reviews which have been posted with the author's consent back to the preprint version. So we are hoping that this is a clearer way of navigating through the different kinds of responses and evaluations that are made to preprints. We've also decided to collect together all the preprints that have peer reviews, author opted in peer reviews.
JOHN INGLIS: And there is a link to that collection on the All Articles page. And then underneath, there are collections of articles by subject category. And you can see off the top there, Animal Behavior and Cognition and Biochemistry. That's just the top of the list of 26 subject categories. And when you go to that collection, you can see-- you can sort by the evaluating entity, as you can see, those MBO-- the Review Commons, eLife, and Peer Richard Science.
JOHN INGLIS: So this area of preprint evaluation is expanding enormously. It has been given a great deal of impetus by the pandemic, and they're setting up of a variety of projects designed to help readers interpret and understand the significance of pandemic-related preprints. And Johnny is going to talk more about the different way in which preprints have been handled by comparison with other kinds of topics.
JOHN INGLIS: So that is my presentation. I look forward to the discussion at the end. And now, my-- I'm going to hand over to Nonia, who is going to take it from here.
NONIA PARIENTE: Let me unmute myself and share my screen. Thank you very much. So-- wait a minute. I'm not sharing the right screen, am I? One second, I have to restart. Share screen, yes. Sorry, where am I? So I am going to give you guys a little vignette about how-- the editorial work plus process with the preprints in this ecosystem of such scholarly publishing.
NONIA PARIENTE: Very briefly, because I assume everybody in this audience will know, but PLOS this is a not-for-profit publisher and an advocacy organization with a mission to accelerate progress in science and medicine. Here is our portfolio, which has very recently grown for the first time in many years in five new journals. So it's the first time I present it complete with PLOS Biology and PLOS Medicine as the flagship journals, and several journals dedicated to publishing sound science with high ethical and technical standards but without considerations of conceptual advance and some community journals in the middle catering to specific communities.
NONIA PARIENTE: So this is an overview of preprint adoption in our published papers in 2020 and 2021. The numbers for PLOS Biology are underestimates because we recently discovered that through a glitch in our system, we actually are missing some that are, for example, being preprinted but submitted to the journal independently of the B2J route that John was speaking of a moment ago.
NONIA PARIENTE: And yeah, and we need to keep in mind when looking at these numbers that, for example, in PLOS One, a lot of the content is out of scope of the most mainstream extreme preprint servers. If we look at our relationship with bioRxiv specifically-- because those were-- The previous table was about papers that had preprint posted anywhere-- these are the papers that either opted in to have PLOS and the positive preprint into bioRxiv, or had already posted a preprint in bioRxiv when arriving at PLOS.
NONIA PARIENTE: And, again, in Bio, there is a slight underestimation of these numbers, I think. And here, you can see a trend over the years. And we're quite happy that in many of the communities, it's clear that our J2B route is actually promoting the deposition of preprints, which, as you'll see at the very end of my presentation, is really one of the organizational objectives of PLOS within the adoption of open science.
NONIA PARIENTE: And so in PLOS neglected tropical diseases and PLOS pathogens, clearly there is an increase in number of authors that have a preprint posted by PLOS after submission of their paper. I'm going to give several vignettes, as I was saying, of different ways in which we've interacted with preprints. We have a really-- we're very big supporters of the preprint movement and are experimenting with different approaches.
NONIA PARIENTE: And so one thing that was not a success story, as John was mentioning in the introduction, was our preprint commenting trial. So for a year, what happened is that comments that were left on preprinted submissions that could be relevant to its scientific evaluation were made available to the handling editors. And these editors-- the idea was that they could use these comments in addition to complementing the peer reviews that they had sourced.
NONIA PARIENTE: We ran this approximately on 1,000 papers across the portfolio for a year only on papers that had opted in to PLOS depositing the preprint for practical reasons of being able to follow the preprints. And what happened was that there were comments, but there were virtually no comments that could inform the peer review process. There were author updates.
NONIA PARIENTE: There were comments saying, this preprint is fantastic or et cetera, but not really critiques of the work or assessments of the work. The ones that were potentially informative often came after peer review had concluded. And this is, again, something I'm going to touch upon in the next slide. It's often that it's coincidental, the posting of the preprint and the submission to a journal.
NONIA PARIENTE: And so if the comment comes substantially after preprint posting then, of course, it won't match with a peer review timing at the journal. And so in the end after one year, we-- PLOS decided that the resources invested in monitoring the comments were not worth the outcome of the trial. Because somebody had to be constantly checking these preprints to see whether there were or not comments.
NONIA PARIENTE: Another thing that we have experimented with is preprint editors. And we've done this in a systematic and in a more ad hoc way. So the systematic way was trialed in PLOS Genetics in 2016. 2016 was relatively early on. And PLOS Genetics wrote an editorial, and they were really pumped up and really excited about the possibility of preprints.
NONIA PARIENTE: And they had a group of associate editors that would look through bioRxiv in their subject area and identify and discuss and invite preprints in a very thorough way. They did this as if these preprints were essentially already at the journal. Several associate editors discussed each one that wanted-- that they were going to call in, et cetera.
NONIA PARIENTE: And I think at the time, the idea was that preprints would be posted not necessarily with a target journal in mind already so that the conversion rates would be relatively high. But what they found and what we now to be the case for a large fraction of authors is that when the preprint was posted, if it had been or it was concomitantly submitted somewhere else. And so they were working and investing time as if the fact of these papers were-- these preprints were submitted to the journal, but then they never came.
NONIA PARIENTE: And some came after a while, but then it was hard to know whether the journal reaching out had an effect on this or they would have come anyway. So the group of-- stopped this official systematic role of doing this across the board a couple of years ago. There are some section editors that do ad hoc preprint calling in, and as do PLOS Biology editors also. And so I have put here for PLOS Biology this ad hoc route is useful for us.
NONIA PARIENTE: I've put here my numbers, so I-- In the last eight months or so, I've called in 50 something, close to 60 preprints with a 61% response rate. So authors do get in touch. Most of them indeed have posted elsewhere. But it's a good strategy for journal outreach nevertheless. And 24% were submitted, which is a very reasonable conversion rate, we think. I do have to note here, though, that I'm a little bit wary-- I mean, I'm happy to do this, and I think it's very useful for the journal from my point of view.
NONIA PARIENTE: But thinking of a future in which perhaps editors would be actively commissioning from preprint servers as the main route of publication, I am really wary of potential biases there. I think that these biases-- the biases everybody talks about in the publication process would likely only increase here. I call in preprints that I see in my social media echo chamber, for example.
NONIA PARIENTE: But we publish a lot of papers that are submitted directly by people having preprints or not, but people that we don't know of work that we are not aware of. And I think yeah, this is a positive thing. This is good. So only direct calling in would make me wary. But anyway, I think it's a good overall initiative. I want to talk a little bit about portable peer review and Review Commons, which is something that John Inglis mentioned, as well as Peer Community In.
NONIA PARIENTE: These are two platforms that PLOS is a member of, and they both provide independent peer review of preprints or of papers that they deposit to a preprint server. And the way this works is they source the reviews in a journal-agnostic way. And then authors can decide to go to any of the journals that are forming part of this platform or this group with those peer reviews.
NONIA PARIENTE: And if it doesn't work at one, then they can try another one. The journals that are members usually reserve the right to add another reviewer if additional expertise is necessary, for example. But with Review Commons, we are seeing more and more that this is less necessary. This is something that started in March 2020. At least we received the first papers in March 2020. And overall, PLOS has received 153 submissions, and we've published 40 papers.
NONIA PARIENTE: Another way that we support the movement, of course, is supporting meta-research on preprints. We support all meta-research and have quite a strong meta-research section. But particularly, we have an increasing number of research on research involving preprints, which is quite interesting. And one of these papers Johnny will talk much more about in a little while, I'm sure.
NONIA PARIENTE: We were asked to explain how COVID has interacted with this whole ecosystem. And for PLOS, there's not really a huge change. Because as I was saying, we were early adopters. And so we already had in place all of these ways of interacting with the preprints. There has been, of course, more volume. More of the preprints that we would have sent to bioRxiv now go to medRxiv, for example.
NONIA PARIENTE: But one thing that was a clear difference is that PLOS signed up to the welcome declaration of research data and finding relevant to COVID. And so what-- beyond just offering the possibility of depositing a preprint to authors whose papers we sent out for review, we strongly encouraged and followed up that they did the positive preprint at the time when a paper was committed to peer review.
NONIA PARIENTE: And we informed the WHO of every submission that went through peer review at PLOS, as was agreed. So I think that was the main kind of operative change that happened. And one last thing I want to talk about is preprints within the PLOS mission. So PLOS has as one of its big four organizational objectives to increase adoption of open science practices throughout the ecosystem, the scholarly communications ecosystem.
NONIA PARIENTE: And within that, increasing the adoption of preprints and increasing trust in preprints is a big part of our effort. We are promoting preprints in our communities as seen by an increased rate of adoption over time. And we're exploring links with other preprint servers. We have very tight links with the Cold Spring Harbor preprint servers that John was talking about. But there is an increasing number, and so we would like to be able to capture the whole preprint ecosystem.
NONIA PARIENTE: And we doing to promote trust in preprints. And one way we're thinking about doing this is surfacing the checks and balances that actually do go into the preprints. One thing I didn't mention but I have thought of is that, for example, in the journal to bioRxiv route, not every paper that wants to be deposited can be deposited. There are preprint fails. And actually, for us, increasingly so.
NONIA PARIENTE: So our preprint checkout rate is around 70% for 2020, down from a higher percentage. Most of the fails have to do with scope issues. But there are checks and balances also in the preprint world, even if they're not peer-reviewed. I think that's the end of my presentation. So if Shalene, you want to go ahead?
SHALENE SINGH-SHEPHERD: Thank you, Nonia. I'm Shalene Singh-Shepherd from Royal Society Publishing. And hopefully, everybody's heard of The Royal Society. But for those who are not familiar with us, we are the UK's independent academy of science based in London. So I am the publishing editor for Proceedings B, which is our flagship biological sciences journal. And we tend to publish content mainly in our core subject areas of ecology, evolution, and behavior.
SHALENE SINGH-SHEPHERD: We were the first channel in our portfolio to appoint a preprint editor in 2017. And that's essentially what I'm going to tell you about today is the whole process we went about in appointing our preprint editor, the processes we use, and also the challenges we've been facing and also some of our successes. Also, another thing to note is that more recently, two of our other journals in the portfolio have also gone down the route of appointing a preprint editor.
SHALENE SINGH-SHEPHERD: And they've also been using similar processes to what we are using. And these two journals are namely Open Biology, which is our online-only fully open access publication that focuses on molecular and cellular biology, and also our sister journal, Biology Letters, which has a similar scope to Proceedings B but publishes short papers.
SHALENE SINGH-SHEPHERD: So we feel that preprint select via archive are a useful resource because they provide a means of providing scientific results to the community very rapidly. And we also feel that from a social justice perspective, they provide a useful means of increasing scholarly productivity, especially for those who may face bias or inequity in the traditional publishing process.
SHALENE SINGH-SHEPHERD: So in 2017, as we noticed preprints were growing in prominence and that more publishers were actually scanning preprints for content, our publishing board took the decision to appoint a preprint editor at Proceedings B. So what we initially did was try to come up with two objectives for the whole initiative to try and see what we would gain from doing such an initiative. And we narrowed it down to we were going to try and raise awareness of the journal across biology, because we are very well known in our three core subject areas of ecology, evolution, and behavior, but not necessarily outside of these areas.
SHALENE SINGH-SHEPHERD: And secondly, what we wanted to do was try to commission more content to try and broaden the scope of Proceedings B, so once again, outside of our core areas. So we were very fortunate that one of our associate editors, Dr. Maureen Nieman from the University of Iowa, volunteered for the role of preprint editor. And bioRxiv was chosen as our server of choice because the subject areas in bioRxiv matched quite neatly with Proceedings B. So what Maureen did was actually turn to a resource in front of her, which was her department of biology at the University of Iowa.
SHALENE SINGH-SHEPHERD: And she actually recruited a preprint team which consisted of graduate students, some of her postdocs, and some undergraduate students. Because she saw this as a perfect opportunity for all of these early-career researchers to interact directly with the scientific process-- sorry, scientific publishing process. So initially, we kicked off in late 2017 with a preprint editorial team headed by Maureen and made up of 20 early-career researchers.
SHALENE SINGH-SHEPHERD: And they came from all areas of biology from the University of Iowa. So this slide just neatly describes the process we are using in Proceedings B. This is obviously evolved over time because from when we started out in 2017 to today, things have obviously changed and we've learned from our mistakes and tried to improve on the process.
SHALENE SINGH-SHEPHERD: So essentially, each of the team members in the preprint team is assigned a particular subject area. And they tend to scan papers in bioRxiv on a monthly basis. Promising papers are then assessed more thoroughly, so they look at the entire paper and not just the abstract. And papers are then selected and added to a database, which the team then meets and evaluates. And Maureen will then make a final assessment as to which of these papers she'd commission to invite for the journal.
SHALENE SINGH-SHEPHERD: We then send out a personalized email to the authors of the paper. And if they choose to accept our invitation, the paper then goes through peer review and hopefully gets published in Proceedings B. How the process has developed over time is that more recently, we've appointed two associate editors within the team. They are still from Maureen's preprint team, but their main role is to decide which of the preprint members would be assigned to which subject categories to handle communications for the team.
SHALENE SINGH-SHEPHERD: And they also handle recruitment. So how have things been going in Proceedings B? The initiative has been running since 2017, and thus far we've had a lot of positive feedback from authors. A lot of them have come back and said to us that they may not have considered Proceedings B as their journal of choice in the past. And they may do so in the future, or they may even come to us if the paper's rejected elsewhere.
SHALENE SINGH-SHEPHERD: But two main things as we have noticed about papers that are already on bioRxiv is that one, they all have a very high standard. But secondly, they have also been submitted to another journal before we approach them. So I think this is just something that other people have seen as well. We have found though that we have been successful in our outreach.
SHALENE SINGH-SHEPHERD: So one of our objectives was to raise awareness of Proceedings B, especially in areas outside of our core. And we have found that this initiative has been very helpful in trying to attract authors who may not have considered us in the past and encouraging them to submit to Proceedings B. In my slide, I've put up some headline stats for you. We've recently evaluated all of the data we have from our preprints initiative.
SHALENE SINGH-SHEPHERD: And this has been analyzed and written up into a research paper, which is currently at review. But what I can share with you today is that to date, we have sent out at least 1,200 invitation emails to authors of bioRxiv papers. From that, we've had over 100 papers submitted to the journal. And we've had about 30% of these papers accepted for publication.
SHALENE SINGH-SHEPHERD: The acceptance rate for the solicitor's papers doesn't differ very much from the acceptance rate of papers traditionally submitted to Proceedings B. But what we have found is that in certain subject areas, these solicited papers have been higher. So if I use the example of two subjects of synthetic biology and bioengineering where Proceedings B traditionally sees a very low number of submissions, the number of papers that have come via the preprint route is far higher.
SHALENE SINGH-SHEPHERD: One other thing that we have done was also to do a survey of the preprint editorial team members, just to get their feedback on how they feel the initiative has been going and also their thoughts on preprints. And what we found that many of the early-career researchers involved have said that this was a great opportunity for them to learn more about the publishing process and to engage with it, and also to learn more about the things like the scope of a journal and what peer review actually entails.
SHALENE SINGH-SHEPHERD: Many have also said to us that they found that their academic career was improved, and it aided their awareness and knowledge of their research area. And for preprints as a whole, they found it a useful resource, as it made science and scientific results available to both scientists and non-scientists.
SHALENE SINGH-SHEPHERD: But I have to say, it hasn't been smooth sailing throughout the whole four years that we've been running our preprints initiative. One of the main problems we found initially was that some authors thought that our invitation email from the preprint editor was spam and from a predatory journal. So we've had to work hard to actually personalize our email just to make it very clear that this is actually from a genuine journal.
SHALENE SINGH-SHEPHERD: We've also had this very awkward issue of we've sometimes invited authors from bioRxiv papers, but it turned out that these authors had already submitted to Proceedings B where they were rejected. So, I mean, that is a very difficult situation to be placed in. So what we do now is make sure we cross-reference all of our invitations against a list of rejected papers already from the journal to ensure that we don't do this again.
SHALENE SINGH-SHEPHERD: Another issue we've faced is journal limitations. So for instance, at Proceedings B, we have a 10-page limit for our articles. And that can be very tricky when we've invited authors from a bioRxiv paper which is far longer, and then we're asking them to reduce the length. Sometimes they're not happy with us asking that. And we've also had to manage author expectations. So essentially, we've had to make it very clear in our invitation letter that we're inviting you to submit the paper.
SHALENE SINGH-SHEPHERD: We'll still go through traditional peer review. And this isn't a guarantee of publication, because we've had a few issues of that misconception in the past. One final thing that we have found is that this whole initiative is trying to broaden the scope of Proceedings B. So we're trying to solicit papers outside of our core areas.
SHALENE SINGH-SHEPHERD: But at the same time, some of these papers that we have invited have been rejected before review by our editorial board members handling them for being out of scope. So this problem is kind of circular in nature. It's very difficult to try and broaden the scope when you don't have the right expertise on the editorial board. So after much discussion, what we decided to do was to introduce a new article type within Proceedings B to accommodate a vast number of papers that our preprint editor had identified that were focused on scientific practices.
SHALENE SINGH-SHEPHERD: In this way, we could then expand our scope to cover that area as well as having more submissions coming in. So what we did in 29 was launch biological science practices as an article type. We have a new associate editor to handle just these submissions. And so far, it's been proving quite popular with, I'd have to say, about 40 to 50 papers already submitted.
SHALENE SINGH-SHEPHERD: And I think to date, we have about seven published. And just finally, I'm going to end on some feedback on what the pandemic has impacted at Proceedings B. So we have seen an unprecedented number of submissions to the journal. I mean, similarly to an increase in papers submitted to bioRxiv, we've also subsequently had more papers being submitted to the journal by this route. We've taken on board the fact that a lot of authors would like a quick transfer of papers, and we've now signed up to the direct transfer from the journal-- sorry, from bioRxiv to the journal.
SHALENE SINGH-SHEPHERD: Especially with COVID papers, we found that authors were very eager to get their publications out as soon as they could, and having a direct transfer has sped up the process a lot more. Our preprint editor also decided to make the preprint editorial team global. So what we did in 2020 was a recruitment drive to try and encourage early career researchers from around the world to join our preprint team.
SHALENE SINGH-SHEPHERD: And to date, we now have 60 members of the team from 29 different institutions and 10 countries across the world. And we're finding this a lot more useful because we have a wider breadth of expertise to now evaluate a wider breadth of submissions. And just one final thing I mentioned at the beginning is that we've expanded now the preprint initiative to other Royal Society journals.
SHALENE SINGH-SHEPHERD: And two of them now, Biology Letters and Open Biology, have also appointed preprint editors. So at this point, I'm going to end there and hand over to our next speaker, Johnny.
JONATHON COATES: OK, thank you. So this story here is actually a really good example of how preprints are beneficial for early-career researchers. I'm a postdoc. I'm a senior also. Hopefully, there's going to be two papers out of this. And I think this is just a really nice example of why all early-career researchers should preprint. So I'm going to put some numbers on a lot of what's being talked about so far, and how the pandemic has affected preprints in particular.
JONATHON COATES: So we know that the pandemic led to an increase in papers. And so we looked at the first 10 months, so from January to October, November time. And in that time period, you can see there's just under 125,000 papers, preprints published in that time frame that are on COVID. And of that, about 25% were preprints, which is a huge number. Normally, preprints represent about 3% of the biomedical literature, which interestingly last year, that went up to 8%.
JONATHON COATES: So there's been a huge impact on preprint usage purely because the pandemic. And what I particularly liked in some of them-- I tried to distill this down to a very small number of slides. But what you're seeing here are on your left, you've got your COVID preprints. And on your right, you've got your non-COVID preprints. The bars that are filled in are those where the senior author has previously posted a preprint.
JONATHON COATES: The bars are not shaded in, or the lighter bars, are those where authors have not previously posted a preprint. So those represent people who are coming to the preprint service for the first time. And what you're seeing across all of the countries we looked at is a significant portion of people turning to preprint service, especially for COVID research, for the first time, which suggests that there's been a radical rethink in how people are approaching sharing their work.
JONATHON COATES: Mainly, I suspect that's driven by the fact that we need to get things out quickly. But there may be other reasons for that. And if we look at-- so this is just from bioRxiv and medRxiv. If you look at our preprint, you'll see that we've looked at other service as well. But bioRxiv and medRxiv are really good examples of preprint servers because they have APIs that you get access really easily to a lot of this underlying data.
JONATHON COATES: So we look at those two servers. What you're seeing here are the abstract views and the PDF downloads for COVID research, which is in red, or in green, the non-COVID preprints. And basically, what we found within that sort of 10-month period is that the COVID preprints are being viewed 18 times and downloaded almost 30 times more than non-COVID preprints were, which is an incredible demand. And that kind of led us to suggest that maybe this is not all coming from scientists, which makes sense if you keep up with the news.
JONATHON COATES: So we looked at where these preprints were being shared. So this panel on the top left here is the percentage of preprints that have more than one citation. And in red, you're seeing COVID. And in green, you're seeing non-COVID. And you can see the stark difference, especially for those preprints that were posted early in the pandemic, so within the first four months.
JONATHON COATES: And although that has decreased, there's still a significant difference at the end of October. This decrease is not just fatigue or anything like that. This is most likely due to our collection window. So we collect this data on the first of November. So those posted in October haven't had time to accumulate citations. We also looked at Twitter. A lot of preprints are tweeted out by bots.
JONATHON COATES: And there's no easy way to filter that out. So what we did was we took all of the preprints that had more than two tweets each. And you're seeing the percentage of that here. So, again, the top line, almost 100% for the entirety of the year is the COVID preprints. And this slightly lower line is the non-COVID preprints. So COVID preprints are being heavily shared across Twitter. And that is not just by scientists, which I'll get to.
JONATHON COATES: We also looked at how news organizations have been using preprints. So you'll see-- and if you look before January actually, you'll see the preprints that are not COVID-- so the bottom line-- generally are not reported on in the news. And I've talked to a number of journalists in doing this work. And generally, the line was journalists do not report preprints.
JONATHON COATES: And you can see that very clearly changed when the pandemic hit, but particularly when the pandemic was kind of really starting to take off. And what you're seeing is that on average, COVID preprints, about 25% of the COVID preprints are reported on in at least one news article, often more than one. And throughout that time period, almost no preprints that are not covered are reported on by news organizations.
JONATHON COATES: The other thing we looked at were whether or not preprints were driving policy changes. I had a strong suspicion early on that they were, but it would be nice to have some data for this. So we looked at the European Center for Disease Control, the World Health Organization scientific briefs, and the UK post based in the UK parliament. They have this service which they write little briefs and they are then used to help inform MPs making decisions and policy-making decisions.
JONATHON COATES: And across those three things that we looked at, you can see COVID preprints are being referenced, not in a huge amount. Sort of roughly 20% of the references in each article were a preprint. But the most interesting part of this data is that when we looked at non-COVID preprints in this time frame, so January to October, all before that, we couldn't find any preprints used in policy documents.
JONATHON COATES: So you're seeing quite a significant shift in how everybody seems to be using preprints for the pandemic. And unfortunately, not all this use is good. So we looked at the top 10 preprinted papers, so COVID and not COVID, and how they were received on Twitter. So what you're seeing here is an example of a preprint that was posted, shared on Twitter, quite widely shared on Twitter.
JONATHON COATES: And I'm just pointing out some of the hashtags we noticed in here. And you see things that are kind of representative of things you would want to be getting across, so disinfect, airborne, social distancing, preprint signs are in there. But what you're seeing are things that the preprint is talking about, and those are getting fairly well delivered across Twitter, at least.
JONATHON COATES: So the scientific message is getting out. On the other hand-- and unfortunately, at the time of this data collection was the second highest tweeted about article-- some preprints are being used by right-wing conspiracy groups, right-wing politicians largely, and a lot of people who are quite racist, unfortunately. And what you see here is that this preprint was, I've said, hijacked.
JONATHON COATES: So you can see hashtags to do with COVID-19 being a lie, fake news, pandemic, ending lockdowns. Bill Gates is in there. Dr. Fauci gets mentioned a lot in these kind of hashtag word clouds. And you're seeing a very clear dichotomy between science getting shared properly and just complete misuse, misunderstanding of what the message should be.
JONATHON COATES: So we really need to think a bit more about how we are actually sharing our work and how we police this. This could have a whole paper on its own. We did not have time to go beyond this, unfortunately. But this really importantly leads to the question of are preprints good enough quality to share? And it's something people have been asking for quite a long time.
JONATHON COATES: And there's been in the past couple of years a few papers that have kind of looked at that topic, but not quite. So we decided to directly address the question of quality between preprint and its published version. We only looked between January and April because this allowed us to look at all of the preprints that had been published that were COVID related, and then also get a date set for a control.
JONATHON COATES: So the first thing we did was we looked at the total number of panels and tables in a preprint or its paper companion. And we just count those up to see if there was a significant amount of extra work being put in there. You can see that the COVID on the left here are lower in terms of the panels and tables than non-COVID, which makes sense because generally, those were also shorter papers.
JONATHON COATES: This is emerging, so you'd expect those to be a bit shorter. But you can see that whether it's COVID or not COVID, between the preprint and the paper, on average there's no difference in the total number of tables and piles. And when we categorized how these were changing where they did change-- so I said over 60% didn't have any changes-- a large portion just had rearrangement, so a figure might be moved from a panel-- from a main figure might be put into a supplementary, for example, or something was moved.
JONATHON COATES: We did notice that roughly 20% had significant content added. So this suggests that there is a very important place for peer review, and that peer review is working and having a positive impact on papers as they progress. And then we also looked at abstracts. And the reason we focused a lot of our analysis on abstract is because we know that a lot of the general public are using preprints and papers. And normally, outside of COVID, a lot of these things are behind paywalls if it's a paper.
JONATHON COATES: And so often, the only thing people will read is the abstract. And the abstract generally distills the key messages of paper. So we reasoned that if the abstract changes, then there's been a key change in one of the conclusions of the paper. We tried a bunch of different automated ways of doing this to try and get some kind of ideas whether the preprint abstract was changing when it was published or not.
JONATHON COATES: So the higher the change ratio for both of these, the more a preprint paper abstract changed. Generally, we didn't see a huge difference, so they were roughly the same. So COVID preprints generally had a slightly higher change in their abstract than non-COVID articles did when they were published. But because these automated methods didn't really line up very well, we decided to do a manual analysis, which I must say, it took a very long time, and I will happily never read another abstract again.
JONATHON COATES: But when we did this, what we found was that for 6% of non-COVID research or 15% of COVID research, the abstracts undergo a discrete change in their key conclusions. This is really important. But if we reframe that, 94% of the non-COVID literature or 85% of COVID literature that were preprints that were then published had no discrete changes in their conclusions, at least within the abstracts.
JONATHON COATES: So this suggests that a lot of the anecdotal evidence you might have with that preprints are generally high quality. This really suggests that they are comparable to the preprint of the published literature. And not that we should treat preprints with the same level that we might treat with the published literature, but that we should treat published literature the same as preprints.
JONATHON COATES: And that is, we should approach both with skepticism and caution. We should read those papers and analyze them ourselves, not just rely on peer review. So we found that the rates of preprint posting were 100 times that of previous epidemics, represent the core of all articles. Authors from all over the world, primarily UK, US, and China-- they were getting viewed and downloaded considerably more than non-COVID preprints were.
JONATHON COATES: And the quality was not detectably different to non-COVID preprints. And we surmise basically that there's been a cultural shift in how preprints are being used that we hope is here to stay. And to end on a nice little positive, a lot of the co-authors on this paper I actually found through Twitter or through just kind of approach that I thought would be really good to get involved with.
JONATHON COATES: And all the authors at the time were early career researchers, which is also I think is a-- just a great thing for a paper. So you can read a lot more about this in our PLOS Biology paper. If you're interested in the changes analysis that we did, that-- we actually submitted that to PLOS Biology with another group from the US fairly recently. But that's currently on bioRxiv. And I have more slides available, a wider slide deck if you want those.
JONATHON COATES: Thank you. Back to you, John.
JOHN SACK: Thanks very much. OK, good. We have-- let's see, 1, 2, 3, 4, 5-- I think six or seven questions to the panelists. So I will try and take us through those. SSP has told us we can go a few minutes over. And we'll use that to try and address all the questions. Let me get back to the first one-- let's see, from Kimberly.
JOHN SACK: Kimberly asks-- and I think this is a-- probably a question for any number of folks. But I think John is probably-- John Inglis has probably handled it before. How is it that an author can select an OA license for a preprint and then assign it-- assign the copyright-- to a publisher for the published version?
JOHN SACK: John? Unmute, John. There we go.
JOHN INGLIS: So if I understand the question, the questioner is saying, is it OK to choose any kind of license on a preprint without regard to the requirements at the journal? And as far as we are concerned, the author owns the copyright on every preprint and chooses the license. And my understanding is that most journals have no problem with transition of a paper from a preprint version to a journal-published version with a different kind of license.
JOHN INGLIS: Shalene is nodding her head, and so thank you for confirming that.
JOHN SACK: There's a related question that I think is pretty quick to answer from Cami. Does a preprint get a DOI, or is a DOI assigned only to the version of record? Why don't we have John take that one as well?
JOHN INGLIS: Every preprint has a DOI, and it is specific to the server. And, again, the convention has arisen that the published version will have the DOI of the journal concerned.
JOHN SACK: So everybody gets a DOI. The next question came from Anne Stone. Hello, Anne. It's for Shalene. Regarding the opportunity to create more equity, has the data shown any impact?
SHALENE SINGH-SHEPHERD: We've analyzed that data. The only difficulty we've had is that' we don't always have information, that more information that we need on authors selected in the submission system that we're using. But that's something we're currently working on at the Royal Society to try and improve on our system before we could give a definitive answer of how much of an impact we've had using this initiative.
JOHN SACK: John Long asks a question I think is also intended for Shalene. How have you gauged enhanced awareness of the journal?
SHALENE SINGH-SHEPHERD: I think at the moment, we're purely doing that on the response rate that we're getting from authors that we solicit. We are collecting information from everyone who actually has come back and responded to our emails. So in the fact that we know we've sent out 1,200 emails, we've also collected data on the number of people who have actually responded. And in that way, we've been measuring the kind of response that we've been getting.
SHALENE SINGH-SHEPHERD: But, of course, not all of the responses we've had have been positive. We have had a few people come back very negatively or who have told us never to approach them again. But we're currently basing data that we collect in that way on how we are measuring whether this outreach has improved for proceedings or not.
JOHN SACK: So I think the next question from George could go to Shalene and Nonia both, if you both commented on this, in talking about preprint editors. The sense of the main reason, do you have a sense of the main reason authors choose not to submit to the journal if they were sent a solicitation email? Is it because it's spam? Is it because it was already submitted somewhere else, or they didn't want to publish the paper in its current form?
JOHN SACK: Nonia, why don't you take that first and then Shalene.
NONIA PARIENTE: So for us, clearly it's because it's somewhere else. I say 70% to 75% of people we approach say the paper is somewhere else, or they are just about to submit it somewhere else. So they deposited a preprint with a very clear, specific place of submission in mind. Having said that, we use portable peer review, for example. So we encourage transfers from other journals with the peer-reviewed source that the other journals-- so we have seen some of those papers eventually make it to us.
NONIA PARIENTE: The fact that it's submitted somewhere else doesn't mean it's going to be successful where it's submitted. But I would say that there are two very clear demographics also for my interactions with authors. There are the people who are very community-build approach. And so they really preprint a paper to get community feedback and improve the paper. Most of those feedback is usually in an email or in a personal communication, but it doesn't mean it doesn't exist.
NONIA PARIENTE: It does. And so there are, I would say, perhaps 25% to 30% of people who actually do it for that. And those people will-- are very open minded about where to ultimately submit. And the others do it almost concomitantly.
SHALENE SINGH-SHEPHERD: I mean, I would have to agree with that and say it's been similar for us that people have already submitted their papers elsewhere by the time our preprint editors approached them. They've come back and said, well, if it's been rejected elsewhere, we may reconsider and submit to you, which for us is very helpful. But I think that's one of the main issues we've run into, as well as the fact that sometimes people interpret our invitation as spam from a predatory journal.
SHALENE SINGH-SHEPHERD: But I think because we've now personalized what we say in those invitations, that has helped us somewhat. But I think yeah, the main problem is still when people post on bioRxiv, the paper is also simultaneously submitted to a journal.
JOHN INGLIS: John, maybe I could just say a couple of years ago, we did a survey of about 4 and 1/2 thousand authors. And we found in our survey that 40% of them said that they simultaneously deposited a manuscript on bioRxiv and submitted to a journal. But 40% of them said they just submitted to the preprint server. So one of the things that we have been thinking about a lot recently is how to allow authors to signal their interest in being approached by editors.
JOHN INGLIS: And that would seem like a useful service for authors. But on the other hand, we are also hearing from authors who get an enormous amount of solicitation from journals that they do not want to hear from. So balancing those two concepts is tricky, and we are still thinking that through.
JOHN SACK: Thanks, John. I think the last question in the list or next to the last question is for Johnny. Have you gone back to the journalist to find out the implications of preprints in reporting, and will they go back to not reporting preprints?
JONATHON COATES: It will be a small sample size. But generally, they seem to quite like preprints. There's no embargo on them. One journalist actually pointed out that he really likes the bioRxiv website because it's laid out so easy for you to pick out what you want and what category you want. So from the people I've spoken to, they will probably be sticking with it.
JOHN INGLIS: Again, John, could I jump in there? I've been talking with journalists too like Johnny. And one thing we have to bear in mind is that when the pandemic hit, most-- even science journalists were unfamiliar with preprints, because they were-- They had been a thing in physics and math. And let's face it. Most science journalists do not report on physics and math. So in the beginning of 2020, there was an enormous amount of confusion about how to handle preprints.
JOHN INGLIS: And I think talking to folks that-- say, The New York Times and other places, they now have really got their methodology down. And they know both of the pluses-- as Johnny said, the pluses are instant access to the latest research-- but also how to assess how other experts think about this particular work. And many journalists have their own sort of little brains trust of people, trusted authorities who they can go to for opinions.
JOHN INGLIS: So I think we're seeing an enormous improvement in how parents are reported, and also an increase in the frequency with which they are noted as not being peer reviewed, which was another thing that didn't happen as much as it should have done in the early days of the pandemic.
JOHN SACK: And a couple more questions just came in. Maybe they're coming in faster than we're answering them. Let's see. Adam has a question for Shalene. Can you share the percent of solicited papers which were ultimately accepted? I think I saw that number.
SHALENE SINGH-SHEPHERD: Up to date, we've had about 30% that were from the preprint streak that we've accepted and published.
JOHN SACK: That's significant. The question is from Sarah.
NONIA PARIENTE: From the ones that were submitted, but I guess the question is from the solicitation, so from the 1,200.
SHALENE SINGH-SHEPHERD: Yeah, we had about over 100 submitted. I think it was about 100 and--
JOHN SACK: Yeah, that's right, 100 and-- yeah, it was about 1 in 120.
SHALENE SINGH-SHEPHERD: Of the 30% were accepted.
JOHN SACK: Question from Sarah. Why don't we send this one to John, John Inglis? Question about where preprint is revised, how are revisions typically tracked? And is a record of the exact revision made available to readers?
JOHN INGLIS: All the revised versions live on the preprint server, so-- and can be looked at separately. We do have a tool for allowing authors to choose to note the changes that have taken place from version to version. It's not something we-- a relatively small proportion of authors actually do that.
JOHN SACK: And final toss-up question for all the panelists who care to comment-- what's the fake news out there about preprints? Are there any misinformation kinds of things that you'd like to correct, especially publishers' perceptions of? Why don't we start John, Nonia, Shalene, and then Johnny.
JOHN INGLIS: Gosh, where to start. That's a-- [LAUGHTER]
JOHN SACK: Choose just one, John. [LAUGHTER]
JOHN INGLIS: OK, well, I think the one I would choose is that this is somehow-- that preprints are second class research outputs. I think the data that Johnny showed and other studies of their comparisons between preprint versions and peer-reviewed versions show that when the adjustments take place, they can be significant. But there is not-- I think the events of the last few years have demonstrated that a preprint version is not some inferior output that can be ignored or should be ignored.
JOHN INGLIS: And I think that that was one perception that we encountered quite a lot at the beginning of bioRxiv.
JOHN SACK: Nonia?
NONIA PARIENTE: I think one of the bigger things I have to dispel all the time is that anything goes, that it's like a dumping ground for anything. It stems from lack of appreciation and knowledge, but also from the fact that, for example, some don't have methods or what have you. And so I really stress the checks and balances. As John was mentioning at the beginning, there are screens. It's not peer review, but it's-- they are screened.
NONIA PARIENTE: And indeed, from having the experience of the J2B workflow, we know that not everything will fit. And it's not that just anything can be there. I wanted to make a very little comment, if I may, on these comparisons of preprints and published versions. And this is semi-anecdotal but true, and I've worked in many publishers over my career. Authors do not revise their abstracts, no matter how much they've changed their paper.
NONIA PARIENTE: This just happens. They throw the title in an abstract two years ago, and that's what gets to pre-publication unless an editor steps in. But in response to Johnny's preprint, I've actually changed our revision letters to stress that fact. But it really depends on how detailed the editorial work is whether or not you would change-- you would see change in an abstract, even if changes have occurred.
JOHN SACK: So I think we're going to get cut off in less than a minute automatically. Shalene, anything on this topic?
SHALENE SINGH-SHEPHERD: I mean, I would just emphasize quality, that quality of preprints is exceptional. And the misconception that, as Nonia said, anything goes, that's clearly not true from what we've been finding.
JOHN SACK: Johnny, last word?
JONATHON COATES: Same thing, that preprints are just as good as papers for the most part, and they're probably the greatest thing at leveling the field a bit for early-career researchers. I could not have done this without that.
JOHN SACK: Thank you all very much. We appreciate the attention of so many attendees and also excellent panelists. Thanks.
JOHN INGLIS: Thank you all.
NONIA PARIENTE: Thank you.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode