Cadmore media player playing video Parallel Session 1a: New Horizons in Open Research: Open peer review
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
Parallel Session 1a: New Horizons in Open Research: Open peer review
Description:
Parallel Session 1a: New Horizons in Open Research: Open peer review
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/1a435d06-d09b-4f76-8a60-ed5701313ec3/videoscrubberimages/Scrubber_20.jpg
Duration:
T00H48M01S
Embed URL:
https://stream.cadmore.media/player/1a435d06-d09b-4f76-8a60-ed5701313ec3
Content URL:
https://asa1cadmoremedia.blob.core.windows.net/asset-fa48d355-29f6-4dff-8458-98e07d6f4af7/Parallel 1a update.m4v
Upload Date:
2020-09-17T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
LIZ ALLEN: OK. Hi everybody. Welcome to this parallel session. This is, again, one of the themes around the New Horizons in Open Research at the conference, and we've got three parallel sessions. This is the first one, and then there's one around open methods which follows this directly. And then there's another session around open books and innovations in open books.
LIZ ALLEN: So this one, we're going to focus on open peer review and examples around some of the work that's being done on open peer review. We've included this session because most of you probably know open peer review is sort of being classed as one of the pillars of open research. So it's kind of one of the areas that is being used increasingly, and there's a lot more awareness around what it is.
LIZ ALLEN: But I think one of the things that is really interesting around open peer review is there are lots of different models of open peer review, and it's kind of a generic umbrella term for lots of different things around open peer review. And I think what we're going to explore today is some of the incarnations of that, and what, how and when different publishers are starting to use open peer review in their processes, and what open means in the different contexts.
LIZ ALLEN: So I've got three speakers today. We're going to do questions at the end because this session is quite limited in time, and I think it could be good to get the speakers to say what they want to say, and then we'll have sessions at the end. And we're going to start. These are our speakers. We're going to start in this order.
LIZ ALLEN: So we're going to have Matthew Hayes, who is director of Publisher and Funder Growth at Publons, and he's going to talk about some practical examples of how Publons are working with publishers to support open peer review in the process. Then we've got Bahar Mehmani, who is the Reviewer Experience at Elsever, and she's done a load of work around perceptions around how peer review is being adopted, and what authors actually think about peer review, and then feeding that into the whole publisher model and how they might use open peer review.
LIZ ALLEN: And then we're going to have a discussion. Jadranka's going to talk about some of the sort of bigger issues around open peer reviews, some of the policy and some of the issues around how it works and what works, and what we should be doing. So I'll start with inviting Matt onto the floor. Oh, it's Jadranka. We're in the opposite order, then.
JADRANKA STOJANOVSKI: Hello, everyone. So yesterday morning, our keynote speaker Stephen mentioned numerous quantitative indicators used, and more often misused, in scholarly communication. And depending on country and/or on institution, criteria for academic career advancement and other kinds of assessments are predominantly based on quantitative indicators. For example, in the country where I'm coming from, Croatia, quantitative indicators are so predominant that actually, peer review is still the only qualitative insight into publication.
JADRANKA STOJANOVSKI: There are many definitions of peer review. And we can agree that actually, peer review is a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the fields. And all definitions agree on the role of peer review.
JADRANKA STOJANOVSKI: It's widely held to be an essential element of scholarly communication. So it's interesting that the studies that peer review is the topic became very popular recently. So there are many, many studies on peer review as a process. And actually, results of the studies show that the peer review, close peer review, traditional peer review, has many disadvantages.
JADRANKA STOJANOVSKI: For example, reviewers don't agree much, and it was analyzed in almost 2000 studies. Reviewers don't reliably select highly cited articles. Peer reviewers are not so effective in detecting errors. That's probably because they usually don't have the research data at disposal. And even peer review that's considered rigorous, it doesn't prevent the publication of fraudulent signs, even in the most prestigious journals.
JADRANKA STOJANOVSKI: It's expensive, biased, unreliable, open to abuse. And of course, it's slow, causes many delays. And last but not least, the reviewer can only advocate for his or her view of the research problem, and may reject the manuscript on a topic that she or she personally deals with.
JADRANKA STOJANOVSKI: And I took as an example also a recent practice of many journals that authors are suggesting to reviewers. And we should really consider and reconsider this practice, because for example, the example of the Journal of Vibration and Control shows how these can be in our [INAUDIBLE] world delicate.
JADRANKA STOJANOVSKI: And last year, they needed to retract 60 papers because of authors actually peer reviewing their own papers. So could open peer review solve some of the issues I mentioned previously? Of course, open peer review is a major pillar of open science.
JADRANKA STOJANOVSKI: And looking for definitions of open peer review, we will see that there is no consensus on what open peer review exactly means. But it's similar with open science. Open peer review, like open science, it's very complex and has, really, many layers. And I will speak very soon about these layers.
JADRANKA STOJANOVSKI: So we asked Croatian editors what they think about open peer review. And it was interesting that the majority of them considered by open peer review, where reviews are publicly available and where authors' responses are also publicly available. Yes, it was also interesting that more than 20% of Croatian editors had no idea what open peer review means exactly.
JADRANKA STOJANOVSKI: So two years ago, Anthony Ross-Hellauer published a paper with seven type of open peer review. And that was very interesting because, actually, he tried to cluster all appearances of open peer review. And he defined the seven types open identities, the authors and reviewers are aware of each other's identity.
JADRANKA STOJANOVSKI: Open reports, of course, their reviews are publicly available. Open participation, the wider community is involved. Open interaction-- actually, there is a continuous discussion between authors and reviewers. And this discussion is encouraged. Open peer review manuscript-- depending on different practices, manuscript could be made immediately available, even before a peer review process.
JADRANKA STOJANOVSKI: And what is more interesting that some journals have practice that even if the article is rejected, as the conclusion of the peer review process, the article remains on the journal's website. Open final-version commenting-- so this is very, yes, promising because tell us that when article is actually published, and we used to call it the final version of the article.
JADRANKA STOJANOVSKI: It's not final. So we are still discussing. And new version could be published too. And interesting, last, open platforms-- of course, peer review as a process is mostly facilitated by journal editors. But the initiatives for review is decoupled from publishing. And there are services offering just peer review.
JADRANKA STOJANOVSKI: So my proposal is because I think that Tony's seven types doesn't cover-- don't cover, sorry-- all appearances of open peer review. So here is my proposal, that we define layers of openness in peer review, which can be combined in many, many different way. So first, we should ask ourselves who opens identity?
JADRANKA STOJANOVSKI: And that's related to the first type, defined by Tony. So it should be disclosed, author identity to reviewer, reviewer to author, or publicly. Member of the community shall be allowed that they are anonymous, for example. What is open? It's very important. So usually, by journal employing open peer review, you can find reviews, you can find author's responses, and you can find comments.
JADRANKA STOJANOVSKI: But actually, we could also include an open, different kind of the communication. And also what should be open is research data. Who is reviewing, and who is commenting? So shall we invite formal peer reviewer, or shall we let potential peer reviewer to choose article and to decide if they want to do peer review or not?
JADRANKA STOJANOVSKI: Should we invite public to do peer review? What is commented could be manuscript reviews. Actually, by journals who are already employing open peer review, usually a manuscript, is commented, but not reviews. And we can comment also. Author responses, final version, research data, everything. Sorry.
JADRANKA STOJANOVSKI: And other layers of openness in peer review includes when peer review takes place-- immediately after submission, or during peer review process, after final version has been published. When is the manuscript or paper available, how its communication process organized, and where the process of peer review takes place-- in journal, or separate platform dedicated for peer review.
JADRANKA STOJANOVSKI: So if we look carefully, we could agree that open peer review have many advantages, and can bring us less subjectivity, more reliable selection, more effective detection of errors, less fraudulent science, and also open review can positively affect reviewer's reputation. But there are also some threats.
JADRANKA STOJANOVSKI: And especially, it's interesting because this is also example from Croatia. Actually 150 editors responded to our survey, and we asked them about possible threats of open peer review. And, actually, when you are part of the small scientific community, you should be very careful, especially with these aspects of disclosing reviewer's identity, because could be very, very tricky in small community.
JADRANKA STOJANOVSKI: So Croatian editors are quite homogeneous in, actually, opinion that open peer review would enhance peer review quality. But it's also obvious that only 7% give the biggest grades to that option. Also, they are quite homogeneous in the last that open peer review would endanger objectivity.
JADRANKA STOJANOVSKI: And yes, of course, it will depend to how we define open peer review and which level of openness will we adopt. And I would like to finish. That's my last slide. Yes, I mentioned numerous studies on peer review and some results of these studies.
JADRANKA STOJANOVSKI: But I would like to emphasize at the end the role of an editor, which is often neglected, even in the studies. So usually, studies on peer review fail to acknowledge the editor as a key agent in the legitimization of peer review process. So I would like to use, also, this transformation from traditional peer review to open peer review, which will happen, towards the system where editors take center role as a curator, not judge or gatekeeper, and where editor is also recognized, and as an editor's important role.
JADRANKA STOJANOVSKI: Thank you.
LIZ ALLEN: K, who's next?
MATTHEW HAYES: Good morning, everyone. Thank you very much, Dranka, you really set me up perfectly there. I was particularly interested to see that editor survey, where you saw that, I think, only 20% of editors favored making reviewer identity public, but over 40% favored making the reports public, which tees me up quite nicely to talk about transparent peer review. In the interest of time, I'm going to skip this bit, because I think, again, Jadranka gave the context to this much better than I can.
MATTHEW HAYES: But yesterday's conversation about transparency and research and, more broadly, the open science movement, I think puts the opening up of peer review into context. And you can see some of the advantages around providing greater visibility and recognition, the potential to advance research integrity and reproducibility, as well as the potential to impede research manipulation and fraudulent peer review, all on the basis that we're shining a light on something that is previously behind closed doors, and providing that accountability and trust.
MATTHEW HAYES: Interest in opening up peer review, which is a deliberate choice of words there, again to indicate how broad the field is. I think it's really growing. And open peer review is just one possibility there. We conducted a number of researcher surveys at leveraging our community of reviewers. We've done them both to publication peer reviewers and grant peer reviews.
MATTHEW HAYES: And I know our focus here is on publication peer review. But you might be interested in some of these findings. We found that 40% of publication peer reviewers believe that more transparency of review identity would have a positive impact on peer review, compared to 60% of grant peer reviewers. And when we dived further into the publication peer review stat, we found a great variance across ages. So we found early career researchers were much more comfortable with the idea of transparent peer review than mature researchers.
MATTHEW HAYES: So I think that's interesting to note in publication peer review and suggests that attitudes are changing. In terms of the grant peer review perspective, I really need to understand that more. But I'm very surprised by it, to be honest, because you normally think of that as a more sensitive area than publication peer review. But anyway, I take from that a positive conclusion that more interest in this field is coming out from researchers.
MATTHEW HAYES: And there's also, of course, interest from publishers. So ASAPbio, the non-profit that's promoting transparency in research, coordinates a letter from publishers, committing, in advance, to opening up their peer review workflows in the coming months and years. A number of prominent publishers signed up to that. However, there are practical difficulties in implementing open peer review workflows, particularly cross established and legacy workflows.
MATTHEW HAYES: And at the moment, we find a number of publishers are using manual workarounds, including involving editor editorial time, which we think could be better spent elsewhere. And we think we have a solution to that. So what is our solution? Well, the idea behind it is to be robust and to be scalable. We think this is only worth doing if this can scale. And we also think that the way of doing that is working within established workflows.
MATTHEW HAYES: So we have done that. And I will show you that in a moment. Some of the main advantages are that it requires minimal work from the publisher. Again, we think that's important to make it efficient and able to scale. It's a simpler, more understandable process for the researchers. Very simple logic of questions that are asked of authors and reviewers.
MATTHEW HAYES: Leveraging our Publons community, we're also putting that transparent peer review content on profiles. So we're trying to make it a recognizable attribute and contribution from the researcher. And finally, of course, in the GDPR world, we have to be very mindful of privacy concerns. And that's something we've also taken into account. And why do we call it transparent peer review?
MATTHEW HAYES: Well, I think that's where we think that open peer review is, of course, about having both the peer review content and the reviewer names published. Within transparent peer reivew, you can just publish the review reports, but keep the identities anonymous. And we think that's one way of gradually introducing this change.
MATTHEW HAYES: So what does the workflow look like? So we've partnered with ScholarOne, a fellow web to science group company and editorial submission system, to do this, and also with Wiley publishers who were our launch partner at the outset. And learning there, I would just say is how valuable it is to work with a publisher that really understands their processes and can help you make tweaks along the way in terms of making it more efficient and scalable.
MATTHEW HAYES: It's a quite simple process, really. On the ScholarOne side of things, in the editorial submission system, we ask all the relevant approvals from the author and the reviewer. And we can amend that, make it flexible, according to how the editor wants to roll it out. For example, if the editor wants to mandate that every author needs to commit upfront to their peer reviews being made public, but give reviewers the choice as to whether to assign their name to it or not.
MATTHEW HAYES: Then the publisher sends us a feed of their accepted articles, which we then go and claim DOIs, so we can make that review content sizable and part of the scholarly record. Then the publisher tells us when the article is published, and we can trigger the publication of that peer reviewed content on the Publons' platform. But this is easier to see in situ. So here it is in situ.
MATTHEW HAYES: So Wiley was our launch partner. We began with one journal in September last year. Here is a transparent peer review article. If you zoom in there, you can see that underneath the title just above the abstract, we have a link out to the Publons' page, where you can find the peer review content. That then takes you to the Publons' website, where you get an article page, which lists all of the review content.
MATTHEW HAYES: Although, I'm sorry about my voice. This is a bit of a cold. I'm sounding very deep. But I think it's quite cool. So you can zoom in here on the contents page. And then you can navigate through to those individual aspects of the review process. So here is one review report early on in the process.
MATTHEW HAYES: And what we think is really exciting is you can probably just make out there, at the bottom of that review, it's been signed by the reviewer. And you can see his name and photo. And that means that he's also a Publons user. So we can then link through to his Publons profile, where you'll notice that under his peer review summary, you can see his open peer reviews. So you can see we're trying to make this a recognizable part of his contribution.
MATTHEW HAYES: And we want to extend that and maybe look at badges and other ways that we can make these champions of transparent peer review real leaders in their field, which I think speaks to more general trends you see in open science. In terms of results, so as I say, we launched this with Wiley last year on one journal. We then extended it to 10 more journals in January this year, particularly looking at other disciplines to try and see what the variants were across the sciences.
MATTHEW HAYES: And we are [INAUDIBLE] Wiley are about to announce the next stage of rollout next week, in Peer Review Week. That's very exciting for us. We've seen it go from one to 10, and two more journals, shortly to be announced. In terms of our findings, it's still early days. I would say in terms of popularity, what's really encouraging is 87% of authors are choosing transparent peer review.
MATTHEW HAYES: Of course, it's not always optional. In some cases, that's mandated by the editor, but other cases not. So that's very encouraging to us. In terms of reviewer acceptances, and whether we see an uptake in acceptances of review invitations or a decline, there's been no meaningful differences, to be completely transparent.
MATTHEW HAYES: We saw a small decrease in some journals, offset by a small increase in others. So we'll keep monitoring that and see if we see any changes there. Likewise, no big difference in the efficiency of the peer review process, which I think you can take as positive at the outset. We're not seeing reviewers slow down their process as a result of participating in this.
MATTHEW HAYES: However, we did see meaningful differences with two journals, where the time from-- sorry, time to initial decision was quicker by around 20 days. So I think we're interested in exploring that further and seeing if this does speed up the process when it is transparent and there's a light being shown on it. Finally, in terms of editor behavior, we noticed that there's greater triaging taking place when you're a transparent peer review journal because you're trying to effectively prune out those manuscripts that you really don't think have a chance of being published.
MATTHEW HAYES: So I think that will be interesting to continue to explore. It suggests some improvements in efficiency as well, and in reviewer workload. In terms of next steps, so we've recently launched with Emerald, IOP, and Royal Society as well. So we're very excited to see how that goes, and we're very keen to explore other partners that want to join us at this early stage as well.
MATTHEW HAYES: And we're learning as we go along. So hopefully you can be a part of that. And finally, I just leave you with this thought that we believe in transparent peer review. It's not just about opening the scholarly record, but actually building on it. So those DOIs, that citable content, and also putting a spotlight on what we think is community-led evaluation, so bringing that recognition to researchers as well.
MATTHEW HAYES: Thank you very much.
BAHAR MEHMANI: Hi, everyone. Thank you. Before I start, and before my slides show up, I would like to invite everyone in the room to join Peer Review Week next week. As the chair of the organizing committee, I'm delighted to announce that the theme is also quite related to the topic of today's discussion. It's quality in peer review.
BAHAR MEHMANI: So you could also use the hashtag quality in peer review when you're tweeting about. And don't forget to follow the Peer Review Week Twitter account for all the exciting events that is planned to go live next week. So what I'm going to present today is about a pilot study that I led in Elsevier back in 2014. And the idea was to publish the reviewer comments alongside the articles on Science Direct.
BAHAR MEHMANI: We piloted with five journals, which were single blind. And with the help of the editor-- of course, we need to have the editor's involvement-- we launched the pilot with-- forget about the slides. I can tell you all the stories. I have all the numbers here as well. So we launched this-- editors wrote editorials on the journal home-- and then also announcement on the journal home page.
BAHAR MEHMANI: There we go. We didn't give any option to authors, simply because we wanted to run this experiment in the simplest way possible. So authors were informed about the journal being participating in this pilot about the open peer review, upon submission, or just looking into [INAUDIBLE] for authors, knowing that the journal is running this.
BAHAR MEHMANI: And reviewers were informed about the pilot. They were not given the option to opt out. When reviewers were invited, they were given the information about the pilot. The journal is publishing peer review reports alongside accepted articles. If you're interested, to join in. You can accept the invitation, and it's up to you to decide to reveal your identity or stay anonymous.
BAHAR MEHMANI: And, of course, if they didn't want to participate, they could just decline the invitation. The only person who could opt out a review or a comment to author would be, then, the editor. Then what would happen was that articles, accepted articles, would go along with the review reports, looking into the reviewer concerns, if they wanted to have their name signed or not. To production, they were typeset in the format of the journal article, and they were just submitted with the DUI, and then be freely available on the article site, with a clear link to the review report.
BAHAR MEHMANI: Well, we did this pilot until 1st of January, 2019, and we collected the data. I just really wanted to test some hypotheses and anecdotes from editors and some researchers about the impacts on the reviewer performance and behavior. And I did this study together with European COST Action. It's called Peere. They are a group of researchers from different parts of the European countries, and some other countries, like in the US.
BAHAR MEHMANI: Researchers interested in the topic of peer review as a research topic itself. So they want to study the peer review as a research subject area. And for that, we need to have data. So that's what we did. I collected all these five pilot journals' historical information, as long as the pilot data points, and along with 65 more journals in Elsevier, with the consent of the editors, to just create a kind of control group to compare the reviewer behavior for these pilot journals in the past and with similar journals in the field.
BAHAR MEHMANI: There we go. The results-- well, so we're talking about almost 10,000 submissions and 20,000 review comments. And we look into this. And we see, first of all, there is a trend in increase of submission, which, again, when you put it in a context, and comparing to the entire data set, you see that this is just the generic trend of growth in science and probably has nothing to do with the pilot itself.
BAHAR MEHMANI: Then we looked into the reviewer acceptance rate. So we checked five things about the reviewer behavio-- reviewer acceptance to invitation letters, the turnaround times, the way they create their recommendations, that they word their recommendations with some sentiment analysis, and also the type of recommendations, decision recommendations that they choose. We also looked into different types and groups of reviewers, in terms of seniority or their agenda.
BAHAR MEHMANI: So if you look into this graph, you'll see that you see a general trend in reviewer acceptance, right? But if you put it in the context again and compare it with the control group, you see no significant decline in the reviewer acceptance rate. We also looked, as I said, into their turnaround time, and we saw that it's slightly more increased in comparison to their historical data on the control group.
BAHAR MEHMANI: But, again, if you put it in the context, these journals received also more and more and more manuscripts. So, again, there is no significant impact on their turnaround times. Next, we look into different groups. We could collect the reviewers' titles, in terms of professor, doctor, or other, if they didn't use any of those.
BAHAR MEHMANI: So we grouped them into professors and early career researchers. And this also confirms what Matthew mentioned-- we could see that the professors were slightly less inclined to accept reviewer invitation during the pilot. But the early career researchers, on the other hand, were really excited about the pilot. And then we checked against gender. We used the Python dictionary of names.
BAHAR MEHMANI: We didn't find any difference between male and female researchers. Then we ran a sentiment analysis to find out how reviewers formulated their comments. We checked them based on the polarity. So polarity means if they are using more positive tones and subjectivity, if are subjective or not, and then, again, comparing with the control group, we found out that actually there is, again, no significant difference between non-participating pilot journals and also the history of the journals and the ones that are piloting the open peer review, in the sense of the publishing comments of the reviewers to authors.
BAHAR MEHMANI: But we noticed that there is a slightly different behavior happening with the male younger reviewers. So they are being more positive, which is a good thing I would say. And I'm not the editor, though. Then we looked into the types of recommendations. So reviewers not only write comments to authors and comments to editors, but they also are requested to choose from different decision types.
BAHAR MEHMANI: And we couldn't find, again, a significant difference between the type of recommendations that they choose. Although, they were given the option to stay anonymous or reveal their identity. So we could see that those who choose to reveal their identity, which is only 8% of those who participated in the pilot, they used more positive recommendation types.
BAHAR MEHMANI: So they were more recommending accept or minor regions. So in conclusion, I could say if you choose to give the kind of authority to reviewers to keep the control on their identities, then you wouldn't find any significant impact on their behavior, at least for the pilot journals that we ran the experiment with. And with that, I can open questions.
BAHAR MEHMANI: [APPLAUSE]
LIZ ALLEN: Thank you. Thank you to the speakers. I think it's really interesting for me to-- particularly, and I'm hoping that all of you as well-- some of this, a lot of myth and hype around open peer review. And it's really nice to have some information and data that kind of questions some of those assumptions that I think a lot of us have had for a while. And it ties very closely to some of the work that Stephen Curry was talking about yesterday.
LIZ ALLEN: Because I think one thing that it would be really interesting to reflect on as well is the incentivization to do peer review. Because one of the reasons for opening peer review is to get credit for the work that you've done, that's traditionally been very hidden. So I think it's really interesting as a whole topic. So can I throw-- I'll throw it to you guys?
LIZ ALLEN: Any questions? Can you say who you are and your affiliation, please?
PHIL HURST: Hello. Phil Hurst with Royal Society. This is a question for the panel. Why do you think younger researchers are more keen on openness in peer review?
MATTHEW HAYES: Yes, please do.
BAHAR MEHMANI: I feel-- I think one of the reasons is that they are looking for getting more credit for what they do. I mean, that's my hypothesis. I need to prove it. But I think when you are young, and you're trying to build your career, you need to get all kind of indicators that you are active in your field, and you do all kind of participation in improving the creation of science.
BAHAR MEHMANI: But when you are a well-established professor, I mean, you don't have to. You don't even have to review, right?
MATTHEW HAYES: Not sure if this-- and it does. I completely agree with what Bahar just said. And I would also add that I think, as well, if you're starting out in research, you can imagine how helpful it is to see peer review content for other submitted articles that have been accepted in your field. So you can get an understanding of, when you're writing a piece, things you need to be mindful of around methodology, structure, et cetera.
MATTHEW HAYES: So maybe there's also more interest for them in generally being able to see how a decision is made about publication when they are setting out on that course themselves, in addition to the recognition, absolutely.
JADRANKA STOJANOVSKI: Yes. Yeah.
LIZ ALLEN: Oh, sorry.
JADRANKA STOJANOVSKI: I think they're not always very keen, as with other aspects of openness-- for example, opening research data. It's, I don't know, maybe some researchers are delighted with opening the data, but has implications. So you are not publishing just your selection of the result and your interpretation, but you are actually disclosing everything what you have done during your research.
JADRANKA STOJANOVSKI: So yeah, but open science-- I think that we are all aware that open science, if you really want to give the size, the value, and the meaning which should have. And I don't think data we should use the term open. Science, by itself, should be open. But if we need to do this, that we can use, reuse, research results.
JADRANKA STOJANOVSKI: So I remember first time when I read that more than 80% of published research results are not reproducible-- sorry-- I was really surprised. So to really put and give the value of science, and to take benefit for the society which is often neglected, and for economy, et cetera, they need to open everything. And open peer review is just part of it.
LIZ ALLEN: The lady in red.
ANDREA POWELL: Hi, it's Andrea Powell from Research for Life. So we know Matthew Publons published a report last year that, in it, was the comment about how underrepresented the Global South is in the peer review system. We also know that publishers are experiencing more moral difficulty getting reviewers to agree to do peer review. So there's an opportunity to extend the network. And I'm just wondering whether an open peer review culture is likely to encourage greater participation from the Global South, or whether you think maybe that there are different cultural challenges that will almost have the opposite effect.
ANDREA POWELL: I don't know if that's something you've looked into at all.
MATTHEW HAYES: I'll just start this by saying that I thought that preLIGHTS yesterday was really cool, and that that maybe is a way of getting more reviewers to participate because it's not as formal as the current peer review process, and you only need to comment on bits of the articles maybe that is relevant to your expertise. You can do it quickly. So maybe that's a way of getting more reviewer participation. I don't feel I know enough to be able to comment on how that is different to the Global North and the Global South.
MATTHEW HAYES: I certainly find, on the Publons' side of things, that-- we talked about this last night, but there's definitely a perception that we need to tackle, about the Global South in participation for journals, you know, discriminating between what is a predatory journal and what is a journal that's just emerging. And I can imagine-- you might see the same-- when a reviewer comments as a non-native speaker, and maybe the science is good.
MATTHEW HAYES: But someone might be dismissive of the way that it's put across. So I would say we need to look at that as well. But--
BAHAR MEHMANI: Well, I can certainly add another angle to that, and that's the mindset of editors or, one step backward, the publishers. So it's important to have a kind of diverse editorial board in place, so that they are aware of the fact that all the manuscripts are going to be reviewed by male, white, American researchers. Well, then you don't have any chance for the Global South, or researchers most broadly. So it's important to have that diversity in the editorial board members and handling editors of a journal so that they can bring in their networks and their understanding of their own background.
BAHAR MEHMANI:
SPEAKER: Hello, my name is [INAUDIBLE].. I have two questions, and one, as a reviewer of the Publons. And I have been wondering if it's totally OK to the journals, which may have blind review system, that I can open my review if I want to. And then I ask [INAUDIBLE] editor of the Finnish Medical Journal, do you find-- we have a double blind peer review-- do you find any need for blinding the author-- I mean, having a double blind peer review-- to avoid gender bias and non-Western bias and all these other possible biases?
SPEAKER:
MATTHEW HAYES: Yes. Shall I take the first part of that? So it was the first part of the question, how difficult is it to move from double blind journal into more open peer review processes? Is that-- my understanding, is that correct? Sorry. Just want to make sure I get it right.
SPEAKER: I may have been unclear. I mean that if a journal has blinded peer review system, is it OK-- I have been wondering if it's OK for the journal that I open my review at Publons' site?
MATTHEW HAYES: Oh, sorry. I see what you mean. I see what you mean. Absolutely, yeah. Well, from the Publons' perspective, practically, we follow the journal policy, whatever that may be. So if it's double blind, even if the reviewer said, I would like to publish my peer review, we would say, this is a journal policy, which I think is only right.
MATTHEW HAYES: But in terms of more broadly, if I can, just answer how you might shift from double blind into open peer review processes, I think maybe this transparency idea where you can keep the identities anonymous is a good way of experimenting, and also making it optional. You can set logic so that both reviewers need to accept, the author needs to accept, and so, in that way, you can have a hybrid journal, where some of the reviews are double blind, some are open.
BAHAR MEHMANI: And I can add to that. There are different studies showing the efficiency of the double blind, in terms of avoiding all kind of biases, with different findings. I mean, there is no unanimous kind of agreement between these studies that it is efficient or not. And it's partly because there is no kind of global study across different disciplines and different journals.
BAHAR MEHMANI: Every journal, now and then, runs a study on its own database to find out if it is efficient or not. Secondly, I think there is an kind of fundamental issue with the double blind, that the editors still is in control. The editor is not blinded to the gender or the background, the author, or reviewers. So there is an inherent bias in there. And then, of course, making things double blind is not easy.
BAHAR MEHMANI: And I don't think there is 100% guarantee that if you doubled blind an author, and the editor wouldn't guess because of all kind of self-citations or the tone or you name it. So--
JADRANKA STOJANOVSKI: [INAUDIBLE]
LIZ ALLEN: We'll have to stop there with the last comment. No, you.
JADRANKA STOJANOVSKI: OK. Yes, just returning to Global South or whatever it means and scientific periphery, or, for example, when I look dissertation in Croatia, we have really rich scholarly publishing tradition, which resulted that today we have a common platform, with more than 450 of fully open access scholarly professional and popular journals.
JADRANKA STOJANOVSKI: And scholarly publishing was never has this commercial aspects in Croatia. So when OPEN Initiative starts, all journals switch immediately, so 12 years ago, to open access. But when we are speaking about peer review, of course you should have in mind that blindness of peer review in such a small scientific community is ridiculous.
JADRANKA STOJANOVSKI: To speak, I know who is author of the paper. It's such a small community. So what Croatian journals are employing, it's mostly so-called double blind to a peer review process. They choose, usually, one reviewer from Croatia, and one from abroad. It depends on the discipline. And I was sure that I will find at least 10 journals two years ago who will be ready to employ open peer review process.
JADRANKA STOJANOVSKI: But I'm still looking, and I still didn't find a journal. It's a very delicate process in the small community, when you really need maybe more transparency than in Western countries. And do you really need this transparency? But you should be very careful because implications could be big.
LIZ ALLEN: K, I think we have to stop there. Sorry everyone's slightly over, but I think there's lots to talk about. Thank you. [APPLAUSE]
Segment:0 .
LIZ ALLEN: OK. Hi everybody. Welcome to this parallel session. This is, again, one of the themes around the New Horizons in Open Research at the conference, and we've got three parallel sessions. This is the first one, and then there's one around open methods which follows this directly. And then there's another session around open books and innovations in open books.
LIZ ALLEN: So this one, we're going to focus on open peer review and examples around some of the work that's being done on open peer review. We've included this session because most of you probably know open peer review is sort of being classed as one of the pillars of open research. So it's kind of one of the areas that is being used increasingly, and there's a lot more awareness around what it is.
LIZ ALLEN: But I think one of the things that is really interesting around open peer review is there are lots of different models of open peer review, and it's kind of a generic umbrella term for lots of different things around open peer review. And I think what we're going to explore today is some of the incarnations of that, and what, how and when different publishers are starting to use open peer review in their processes, and what open means in the different contexts.
LIZ ALLEN: So I've got three speakers today. We're going to do questions at the end because this session is quite limited in time, and I think it could be good to get the speakers to say what they want to say, and then we'll have sessions at the end. And we're going to start. These are our speakers. We're going to start in this order.
LIZ ALLEN: So we're going to have Matthew Hayes, who is director of Publisher and Funder Growth at Publons, and he's going to talk about some practical examples of how Publons are working with publishers to support open peer review in the process. Then we've got Bahar Mehmani, who is the Reviewer Experience at Elsever, and she's done a load of work around perceptions around how peer review is being adopted, and what authors actually think about peer review, and then feeding that into the whole publisher model and how they might use open peer review.
LIZ ALLEN: And then we're going to have a discussion. Jadranka's going to talk about some of the sort of bigger issues around open peer reviews, some of the policy and some of the issues around how it works and what works, and what we should be doing. So I'll start with inviting Matt onto the floor. Oh, it's Jadranka. We're in the opposite order, then.
JADRANKA STOJANOVSKI: Hello, everyone. So yesterday morning, our keynote speaker Stephen mentioned numerous quantitative indicators used, and more often misused, in scholarly communication. And depending on country and/or on institution, criteria for academic career advancement and other kinds of assessments are predominantly based on quantitative indicators. For example, in the country where I'm coming from, Croatia, quantitative indicators are so predominant that actually, peer review is still the only qualitative insight into publication.
JADRANKA STOJANOVSKI: There are many definitions of peer review. And we can agree that actually, peer review is a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the fields. And all definitions agree on the role of peer review.
JADRANKA STOJANOVSKI: It's widely held to be an essential element of scholarly communication. So it's interesting that the studies that peer review is the topic became very popular recently. So there are many, many studies on peer review as a process. And actually, results of the studies show that the peer review, close peer review, traditional peer review, has many disadvantages.
JADRANKA STOJANOVSKI: For example, reviewers don't agree much, and it was analyzed in almost 2000 studies. Reviewers don't reliably select highly cited articles. Peer reviewers are not so effective in detecting errors. That's probably because they usually don't have the research data at disposal. And even peer review that's considered rigorous, it doesn't prevent the publication of fraudulent signs, even in the most prestigious journals.
JADRANKA STOJANOVSKI: It's expensive, biased, unreliable, open to abuse. And of course, it's slow, causes many delays. And last but not least, the reviewer can only advocate for his or her view of the research problem, and may reject the manuscript on a topic that she or she personally deals with.
JADRANKA STOJANOVSKI: And I took as an example also a recent practice of many journals that authors are suggesting to reviewers. And we should really consider and reconsider this practice, because for example, the example of the Journal of Vibration and Control shows how these can be in our [INAUDIBLE] world delicate.
JADRANKA STOJANOVSKI: And last year, they needed to retract 60 papers because of authors actually peer reviewing their own papers. So could open peer review solve some of the issues I mentioned previously? Of course, open peer review is a major pillar of open science.
JADRANKA STOJANOVSKI: And looking for definitions of open peer review, we will see that there is no consensus on what open peer review exactly means. But it's similar with open science. Open peer review, like open science, it's very complex and has, really, many layers. And I will speak very soon about these layers.
JADRANKA STOJANOVSKI: So we asked Croatian editors what they think about open peer review. And it was interesting that the majority of them considered by open peer review, where reviews are publicly available and where authors' responses are also publicly available. Yes, it was also interesting that more than 20% of Croatian editors had no idea what open peer review means exactly.
JADRANKA STOJANOVSKI: So two years ago, Anthony Ross-Hellauer published a paper with seven type of open peer review. And that was very interesting because, actually, he tried to cluster all appearances of open peer review. And he defined the seven types open identities, the authors and reviewers are aware of each other's identity.
JADRANKA STOJANOVSKI: Open reports, of course, their reviews are publicly available. Open participation, the wider community is involved. Open interaction-- actually, there is a continuous discussion between authors and reviewers. And this discussion is encouraged. Open peer review manuscript-- depending on different practices, manuscript could be made immediately available, even before a peer review process.
JADRANKA STOJANOVSKI: And what is more interesting that some journals have practice that even if the article is rejected, as the conclusion of the peer review process, the article remains on the journal's website. Open final-version commenting-- so this is very, yes, promising because tell us that when article is actually published, and we used to call it the final version of the article.
JADRANKA STOJANOVSKI: It's not final. So we are still discussing. And new version could be published too. And interesting, last, open platforms-- of course, peer review as a process is mostly facilitated by journal editors. But the initiatives for review is decoupled from publishing. And there are services offering just peer review.
JADRANKA STOJANOVSKI: So my proposal is because I think that Tony's seven types doesn't cover-- don't cover, sorry-- all appearances of open peer review. So here is my proposal, that we define layers of openness in peer review, which can be combined in many, many different way. So first, we should ask ourselves who opens identity?
JADRANKA STOJANOVSKI: And that's related to the first type, defined by Tony. So it should be disclosed, author identity to reviewer, reviewer to author, or publicly. Member of the community shall be allowed that they are anonymous, for example. What is open? It's very important. So usually, by journal employing open peer review, you can find reviews, you can find author's responses, and you can find comments.
JADRANKA STOJANOVSKI: But actually, we could also include an open, different kind of the communication. And also what should be open is research data. Who is reviewing, and who is commenting? So shall we invite formal peer reviewer, or shall we let potential peer reviewer to choose article and to decide if they want to do peer review or not?
JADRANKA STOJANOVSKI: Should we invite public to do peer review? What is commented could be manuscript reviews. Actually, by journals who are already employing open peer review, usually a manuscript, is commented, but not reviews. And we can comment also. Author responses, final version, research data, everything. Sorry.
JADRANKA STOJANOVSKI: And other layers of openness in peer review includes when peer review takes place-- immediately after submission, or during peer review process, after final version has been published. When is the manuscript or paper available, how its communication process organized, and where the process of peer review takes place-- in journal, or separate platform dedicated for peer review.
JADRANKA STOJANOVSKI: So if we look carefully, we could agree that open peer review have many advantages, and can bring us less subjectivity, more reliable selection, more effective detection of errors, less fraudulent science, and also open review can positively affect reviewer's reputation. But there are also some threats.
JADRANKA STOJANOVSKI: And especially, it's interesting because this is also example from Croatia. Actually 150 editors responded to our survey, and we asked them about possible threats of open peer review. And, actually, when you are part of the small scientific community, you should be very careful, especially with these aspects of disclosing reviewer's identity, because could be very, very tricky in small community.
JADRANKA STOJANOVSKI: So Croatian editors are quite homogeneous in, actually, opinion that open peer review would enhance peer review quality. But it's also obvious that only 7% give the biggest grades to that option. Also, they are quite homogeneous in the last that open peer review would endanger objectivity.
JADRANKA STOJANOVSKI: And yes, of course, it will depend to how we define open peer review and which level of openness will we adopt. And I would like to finish. That's my last slide. Yes, I mentioned numerous studies on peer review and some results of these studies.
JADRANKA STOJANOVSKI: But I would like to emphasize at the end the role of an editor, which is often neglected, even in the studies. So usually, studies on peer review fail to acknowledge the editor as a key agent in the legitimization of peer review process. So I would like to use, also, this transformation from traditional peer review to open peer review, which will happen, towards the system where editors take center role as a curator, not judge or gatekeeper, and where editor is also recognized, and as an editor's important role.
JADRANKA STOJANOVSKI: Thank you.
LIZ ALLEN: K, who's next?
MATTHEW HAYES: Good morning, everyone. Thank you very much, Dranka, you really set me up perfectly there. I was particularly interested to see that editor survey, where you saw that, I think, only 20% of editors favored making reviewer identity public, but over 40% favored making the reports public, which tees me up quite nicely to talk about transparent peer review. In the interest of time, I'm going to skip this bit, because I think, again, Jadranka gave the context to this much better than I can.
MATTHEW HAYES: But yesterday's conversation about transparency and research and, more broadly, the open science movement, I think puts the opening up of peer review into context. And you can see some of the advantages around providing greater visibility and recognition, the potential to advance research integrity and reproducibility, as well as the potential to impede research manipulation and fraudulent peer review, all on the basis that we're shining a light on something that is previously behind closed doors, and providing that accountability and trust.
MATTHEW HAYES: Interest in opening up peer review, which is a deliberate choice of words there, again to indicate how broad the field is. I think it's really growing. And open peer review is just one possibility there. We conducted a number of researcher surveys at leveraging our community of reviewers. We've done them both to publication peer reviewers and grant peer reviews.
MATTHEW HAYES: And I know our focus here is on publication peer review. But you might be interested in some of these findings. We found that 40% of publication peer reviewers believe that more transparency of review identity would have a positive impact on peer review, compared to 60% of grant peer reviewers. And when we dived further into the publication peer review stat, we found a great variance across ages. So we found early career researchers were much more comfortable with the idea of transparent peer review than mature researchers.
MATTHEW HAYES: So I think that's interesting to note in publication peer review and suggests that attitudes are changing. In terms of the grant peer review perspective, I really need to understand that more. But I'm very surprised by it, to be honest, because you normally think of that as a more sensitive area than publication peer review. But anyway, I take from that a positive conclusion that more interest in this field is coming out from researchers.
MATTHEW HAYES: And there's also, of course, interest from publishers. So ASAPbio, the non-profit that's promoting transparency in research, coordinates a letter from publishers, committing, in advance, to opening up their peer review workflows in the coming months and years. A number of prominent publishers signed up to that. However, there are practical difficulties in implementing open peer review workflows, particularly cross established and legacy workflows.
MATTHEW HAYES: And at the moment, we find a number of publishers are using manual workarounds, including involving editor editorial time, which we think could be better spent elsewhere. And we think we have a solution to that. So what is our solution? Well, the idea behind it is to be robust and to be scalable. We think this is only worth doing if this can scale. And we also think that the way of doing that is working within established workflows.
MATTHEW HAYES: So we have done that. And I will show you that in a moment. Some of the main advantages are that it requires minimal work from the publisher. Again, we think that's important to make it efficient and able to scale. It's a simpler, more understandable process for the researchers. Very simple logic of questions that are asked of authors and reviewers.
MATTHEW HAYES: Leveraging our Publons community, we're also putting that transparent peer review content on profiles. So we're trying to make it a recognizable attribute and contribution from the researcher. And finally, of course, in the GDPR world, we have to be very mindful of privacy concerns. And that's something we've also taken into account. And why do we call it transparent peer review?
MATTHEW HAYES: Well, I think that's where we think that open peer review is, of course, about having both the peer review content and the reviewer names published. Within transparent peer reivew, you can just publish the review reports, but keep the identities anonymous. And we think that's one way of gradually introducing this change.
MATTHEW HAYES: So what does the workflow look like? So we've partnered with ScholarOne, a fellow web to science group company and editorial submission system, to do this, and also with Wiley publishers who were our launch partner at the outset. And learning there, I would just say is how valuable it is to work with a publisher that really understands their processes and can help you make tweaks along the way in terms of making it more efficient and scalable.
MATTHEW HAYES: It's a quite simple process, really. On the ScholarOne side of things, in the editorial submission system, we ask all the relevant approvals from the author and the reviewer. And we can amend that, make it flexible, according to how the editor wants to roll it out. For example, if the editor wants to mandate that every author needs to commit upfront to their peer reviews being made public, but give reviewers the choice as to whether to assign their name to it or not.
MATTHEW HAYES: Then the publisher sends us a feed of their accepted articles, which we then go and claim DOIs, so we can make that review content sizable and part of the scholarly record. Then the publisher tells us when the article is published, and we can trigger the publication of that peer reviewed content on the Publons' platform. But this is easier to see in situ. So here it is in situ.
MATTHEW HAYES: So Wiley was our launch partner. We began with one journal in September last year. Here is a transparent peer review article. If you zoom in there, you can see that underneath the title just above the abstract, we have a link out to the Publons' page, where you can find the peer review content. That then takes you to the Publons' website, where you get an article page, which lists all of the review content.
MATTHEW HAYES: Although, I'm sorry about my voice. This is a bit of a cold. I'm sounding very deep. But I think it's quite cool. So you can zoom in here on the contents page. And then you can navigate through to those individual aspects of the review process. So here is one review report early on in the process.
MATTHEW HAYES: And what we think is really exciting is you can probably just make out there, at the bottom of that review, it's been signed by the reviewer. And you can see his name and photo. And that means that he's also a Publons user. So we can then link through to his Publons profile, where you'll notice that under his peer review summary, you can see his open peer reviews. So you can see we're trying to make this a recognizable part of his contribution.
MATTHEW HAYES: And we want to extend that and maybe look at badges and other ways that we can make these champions of transparent peer review real leaders in their field, which I think speaks to more general trends you see in open science. In terms of results, so as I say, we launched this with Wiley last year on one journal. We then extended it to 10 more journals in January this year, particularly looking at other disciplines to try and see what the variants were across the sciences.
MATTHEW HAYES: And we are [INAUDIBLE] Wiley are about to announce the next stage of rollout next week, in Peer Review Week. That's very exciting for us. We've seen it go from one to 10, and two more journals, shortly to be announced. In terms of our findings, it's still early days. I would say in terms of popularity, what's really encouraging is 87% of authors are choosing transparent peer review.
MATTHEW HAYES: Of course, it's not always optional. In some cases, that's mandated by the editor, but other cases not. So that's very encouraging to us. In terms of reviewer acceptances, and whether we see an uptake in acceptances of review invitations or a decline, there's been no meaningful differences, to be completely transparent.
MATTHEW HAYES: We saw a small decrease in some journals, offset by a small increase in others. So we'll keep monitoring that and see if we see any changes there. Likewise, no big difference in the efficiency of the peer review process, which I think you can take as positive at the outset. We're not seeing reviewers slow down their process as a result of participating in this.
MATTHEW HAYES: However, we did see meaningful differences with two journals, where the time from-- sorry, time to initial decision was quicker by around 20 days. So I think we're interested in exploring that further and seeing if this does speed up the process when it is transparent and there's a light being shown on it. Finally, in terms of editor behavior, we noticed that there's greater triaging taking place when you're a transparent peer review journal because you're trying to effectively prune out those manuscripts that you really don't think have a chance of being published.
MATTHEW HAYES: So I think that will be interesting to continue to explore. It suggests some improvements in efficiency as well, and in reviewer workload. In terms of next steps, so we've recently launched with Emerald, IOP, and Royal Society as well. So we're very excited to see how that goes, and we're very keen to explore other partners that want to join us at this early stage as well.
MATTHEW HAYES: And we're learning as we go along. So hopefully you can be a part of that. And finally, I just leave you with this thought that we believe in transparent peer review. It's not just about opening the scholarly record, but actually building on it. So those DOIs, that citable content, and also putting a spotlight on what we think is community-led evaluation, so bringing that recognition to researchers as well.
MATTHEW HAYES: Thank you very much.
BAHAR MEHMANI: Hi, everyone. Thank you. Before I start, and before my slides show up, I would like to invite everyone in the room to join Peer Review Week next week. As the chair of the organizing committee, I'm delighted to announce that the theme is also quite related to the topic of today's discussion. It's quality in peer review.
BAHAR MEHMANI: So you could also use the hashtag quality in peer review when you're tweeting about. And don't forget to follow the Peer Review Week Twitter account for all the exciting events that is planned to go live next week. So what I'm going to present today is about a pilot study that I led in Elsevier back in 2014. And the idea was to publish the reviewer comments alongside the articles on Science Direct.
BAHAR MEHMANI: We piloted with five journals, which were single blind. And with the help of the editor-- of course, we need to have the editor's involvement-- we launched the pilot with-- forget about the slides. I can tell you all the stories. I have all the numbers here as well. So we launched this-- editors wrote editorials on the journal home-- and then also announcement on the journal home page.
BAHAR MEHMANI: There we go. We didn't give any option to authors, simply because we wanted to run this experiment in the simplest way possible. So authors were informed about the journal being participating in this pilot about the open peer review, upon submission, or just looking into [INAUDIBLE] for authors, knowing that the journal is running this.
BAHAR MEHMANI: And reviewers were informed about the pilot. They were not given the option to opt out. When reviewers were invited, they were given the information about the pilot. The journal is publishing peer review reports alongside accepted articles. If you're interested, to join in. You can accept the invitation, and it's up to you to decide to reveal your identity or stay anonymous.
BAHAR MEHMANI: And, of course, if they didn't want to participate, they could just decline the invitation. The only person who could opt out a review or a comment to author would be, then, the editor. Then what would happen was that articles, accepted articles, would go along with the review reports, looking into the reviewer concerns, if they wanted to have their name signed or not. To production, they were typeset in the format of the journal article, and they were just submitted with the DUI, and then be freely available on the article site, with a clear link to the review report.
BAHAR MEHMANI: Well, we did this pilot until 1st of January, 2019, and we collected the data. I just really wanted to test some hypotheses and anecdotes from editors and some researchers about the impacts on the reviewer performance and behavior. And I did this study together with European COST Action. It's called Peere. They are a group of researchers from different parts of the European countries, and some other countries, like in the US.
BAHAR MEHMANI: Researchers interested in the topic of peer review as a research topic itself. So they want to study the peer review as a research subject area. And for that, we need to have data. So that's what we did. I collected all these five pilot journals' historical information, as long as the pilot data points, and along with 65 more journals in Elsevier, with the consent of the editors, to just create a kind of control group to compare the reviewer behavior for these pilot journals in the past and with similar journals in the field.
BAHAR MEHMANI: There we go. The results-- well, so we're talking about almost 10,000 submissions and 20,000 review comments. And we look into this. And we see, first of all, there is a trend in increase of submission, which, again, when you put it in a context, and comparing to the entire data set, you see that this is just the generic trend of growth in science and probably has nothing to do with the pilot itself.
BAHAR MEHMANI: Then we looked into the reviewer acceptance rate. So we checked five things about the reviewer behavio-- reviewer acceptance to invitation letters, the turnaround times, the way they create their recommendations, that they word their recommendations with some sentiment analysis, and also the type of recommendations, decision recommendations that they choose. We also looked into different types and groups of reviewers, in terms of seniority or their agenda.
BAHAR MEHMANI: So if you look into this graph, you'll see that you see a general trend in reviewer acceptance, right? But if you put it in the context again and compare it with the control group, you see no significant decline in the reviewer acceptance rate. We also looked, as I said, into their turnaround time, and we saw that it's slightly more increased in comparison to their historical data on the control group.
BAHAR MEHMANI: But, again, if you put it in the context, these journals received also more and more and more manuscripts. So, again, there is no significant impact on their turnaround times. Next, we look into different groups. We could collect the reviewers' titles, in terms of professor, doctor, or other, if they didn't use any of those.
BAHAR MEHMANI: So we grouped them into professors and early career researchers. And this also confirms what Matthew mentioned-- we could see that the professors were slightly less inclined to accept reviewer invitation during the pilot. But the early career researchers, on the other hand, were really excited about the pilot. And then we checked against gender. We used the Python dictionary of names.
BAHAR MEHMANI: We didn't find any difference between male and female researchers. Then we ran a sentiment analysis to find out how reviewers formulated their comments. We checked them based on the polarity. So polarity means if they are using more positive tones and subjectivity, if are subjective or not, and then, again, comparing with the control group, we found out that actually there is, again, no significant difference between non-participating pilot journals and also the history of the journals and the ones that are piloting the open peer review, in the sense of the publishing comments of the reviewers to authors.
BAHAR MEHMANI: But we noticed that there is a slightly different behavior happening with the male younger reviewers. So they are being more positive, which is a good thing I would say. And I'm not the editor, though. Then we looked into the types of recommendations. So reviewers not only write comments to authors and comments to editors, but they also are requested to choose from different decision types.
BAHAR MEHMANI: And we couldn't find, again, a significant difference between the type of recommendations that they choose. Although, they were given the option to stay anonymous or reveal their identity. So we could see that those who choose to reveal their identity, which is only 8% of those who participated in the pilot, they used more positive recommendation types.
BAHAR MEHMANI: So they were more recommending accept or minor regions. So in conclusion, I could say if you choose to give the kind of authority to reviewers to keep the control on their identities, then you wouldn't find any significant impact on their behavior, at least for the pilot journals that we ran the experiment with. And with that, I can open questions.
BAHAR MEHMANI: [APPLAUSE]
LIZ ALLEN: Thank you. Thank you to the speakers. I think it's really interesting for me to-- particularly, and I'm hoping that all of you as well-- some of this, a lot of myth and hype around open peer review. And it's really nice to have some information and data that kind of questions some of those assumptions that I think a lot of us have had for a while. And it ties very closely to some of the work that Stephen Curry was talking about yesterday.
LIZ ALLEN: Because I think one thing that it would be really interesting to reflect on as well is the incentivization to do peer review. Because one of the reasons for opening peer review is to get credit for the work that you've done, that's traditionally been very hidden. So I think it's really interesting as a whole topic. So can I throw-- I'll throw it to you guys?
LIZ ALLEN: Any questions? Can you say who you are and your affiliation, please?
PHIL HURST: Hello. Phil Hurst with Royal Society. This is a question for the panel. Why do you think younger researchers are more keen on openness in peer review?
MATTHEW HAYES: Yes, please do.
BAHAR MEHMANI: I feel-- I think one of the reasons is that they are looking for getting more credit for what they do. I mean, that's my hypothesis. I need to prove it. But I think when you are young, and you're trying to build your career, you need to get all kind of indicators that you are active in your field, and you do all kind of participation in improving the creation of science.
BAHAR MEHMANI: But when you are a well-established professor, I mean, you don't have to. You don't even have to review, right?
MATTHEW HAYES: Not sure if this-- and it does. I completely agree with what Bahar just said. And I would also add that I think, as well, if you're starting out in research, you can imagine how helpful it is to see peer review content for other submitted articles that have been accepted in your field. So you can get an understanding of, when you're writing a piece, things you need to be mindful of around methodology, structure, et cetera.
MATTHEW HAYES: So maybe there's also more interest for them in generally being able to see how a decision is made about publication when they are setting out on that course themselves, in addition to the recognition, absolutely.
JADRANKA STOJANOVSKI: Yes. Yeah.
LIZ ALLEN: Oh, sorry.
JADRANKA STOJANOVSKI: I think they're not always very keen, as with other aspects of openness-- for example, opening research data. It's, I don't know, maybe some researchers are delighted with opening the data, but has implications. So you are not publishing just your selection of the result and your interpretation, but you are actually disclosing everything what you have done during your research.
JADRANKA STOJANOVSKI: So yeah, but open science-- I think that we are all aware that open science, if you really want to give the size, the value, and the meaning which should have. And I don't think data we should use the term open. Science, by itself, should be open. But if we need to do this, that we can use, reuse, research results.
JADRANKA STOJANOVSKI: So I remember first time when I read that more than 80% of published research results are not reproducible-- sorry-- I was really surprised. So to really put and give the value of science, and to take benefit for the society which is often neglected, and for economy, et cetera, they need to open everything. And open peer review is just part of it.
LIZ ALLEN: The lady in red.
ANDREA POWELL: Hi, it's Andrea Powell from Research for Life. So we know Matthew Publons published a report last year that, in it, was the comment about how underrepresented the Global South is in the peer review system. We also know that publishers are experiencing more moral difficulty getting reviewers to agree to do peer review. So there's an opportunity to extend the network. And I'm just wondering whether an open peer review culture is likely to encourage greater participation from the Global South, or whether you think maybe that there are different cultural challenges that will almost have the opposite effect.
ANDREA POWELL: I don't know if that's something you've looked into at all.
MATTHEW HAYES: I'll just start this by saying that I thought that preLIGHTS yesterday was really cool, and that that maybe is a way of getting more reviewers to participate because it's not as formal as the current peer review process, and you only need to comment on bits of the articles maybe that is relevant to your expertise. You can do it quickly. So maybe that's a way of getting more reviewer participation. I don't feel I know enough to be able to comment on how that is different to the Global North and the Global South.
MATTHEW HAYES: I certainly find, on the Publons' side of things, that-- we talked about this last night, but there's definitely a perception that we need to tackle, about the Global South in participation for journals, you know, discriminating between what is a predatory journal and what is a journal that's just emerging. And I can imagine-- you might see the same-- when a reviewer comments as a non-native speaker, and maybe the science is good.
MATTHEW HAYES: But someone might be dismissive of the way that it's put across. So I would say we need to look at that as well. But--
BAHAR MEHMANI: Well, I can certainly add another angle to that, and that's the mindset of editors or, one step backward, the publishers. So it's important to have a kind of diverse editorial board in place, so that they are aware of the fact that all the manuscripts are going to be reviewed by male, white, American researchers. Well, then you don't have any chance for the Global South, or researchers most broadly. So it's important to have that diversity in the editorial board members and handling editors of a journal so that they can bring in their networks and their understanding of their own background.
BAHAR MEHMANI:
SPEAKER: Hello, my name is [INAUDIBLE].. I have two questions, and one, as a reviewer of the Publons. And I have been wondering if it's totally OK to the journals, which may have blind review system, that I can open my review if I want to. And then I ask [INAUDIBLE] editor of the Finnish Medical Journal, do you find-- we have a double blind peer review-- do you find any need for blinding the author-- I mean, having a double blind peer review-- to avoid gender bias and non-Western bias and all these other possible biases?
SPEAKER:
MATTHEW HAYES: Yes. Shall I take the first part of that? So it was the first part of the question, how difficult is it to move from double blind journal into more open peer review processes? Is that-- my understanding, is that correct? Sorry. Just want to make sure I get it right.
SPEAKER: I may have been unclear. I mean that if a journal has blinded peer review system, is it OK-- I have been wondering if it's OK for the journal that I open my review at Publons' site?
MATTHEW HAYES: Oh, sorry. I see what you mean. I see what you mean. Absolutely, yeah. Well, from the Publons' perspective, practically, we follow the journal policy, whatever that may be. So if it's double blind, even if the reviewer said, I would like to publish my peer review, we would say, this is a journal policy, which I think is only right.
MATTHEW HAYES: But in terms of more broadly, if I can, just answer how you might shift from double blind into open peer review processes, I think maybe this transparency idea where you can keep the identities anonymous is a good way of experimenting, and also making it optional. You can set logic so that both reviewers need to accept, the author needs to accept, and so, in that way, you can have a hybrid journal, where some of the reviews are double blind, some are open.
BAHAR MEHMANI: And I can add to that. There are different studies showing the efficiency of the double blind, in terms of avoiding all kind of biases, with different findings. I mean, there is no unanimous kind of agreement between these studies that it is efficient or not. And it's partly because there is no kind of global study across different disciplines and different journals.
BAHAR MEHMANI: Every journal, now and then, runs a study on its own database to find out if it is efficient or not. Secondly, I think there is an kind of fundamental issue with the double blind, that the editors still is in control. The editor is not blinded to the gender or the background, the author, or reviewers. So there is an inherent bias in there. And then, of course, making things double blind is not easy.
BAHAR MEHMANI: And I don't think there is 100% guarantee that if you doubled blind an author, and the editor wouldn't guess because of all kind of self-citations or the tone or you name it. So--
JADRANKA STOJANOVSKI: [INAUDIBLE]
LIZ ALLEN: We'll have to stop there with the last comment. No, you.
JADRANKA STOJANOVSKI: OK. Yes, just returning to Global South or whatever it means and scientific periphery, or, for example, when I look dissertation in Croatia, we have really rich scholarly publishing tradition, which resulted that today we have a common platform, with more than 450 of fully open access scholarly professional and popular journals.
JADRANKA STOJANOVSKI: And scholarly publishing was never has this commercial aspects in Croatia. So when OPEN Initiative starts, all journals switch immediately, so 12 years ago, to open access. But when we are speaking about peer review, of course you should have in mind that blindness of peer review in such a small scientific community is ridiculous.
JADRANKA STOJANOVSKI: To speak, I know who is author of the paper. It's such a small community. So what Croatian journals are employing, it's mostly so-called double blind to a peer review process. They choose, usually, one reviewer from Croatia, and one from abroad. It depends on the discipline. And I was sure that I will find at least 10 journals two years ago who will be ready to employ open peer review process.
JADRANKA STOJANOVSKI: But I'm still looking, and I still didn't find a journal. It's a very delicate process in the small community, when you really need maybe more transparency than in Western countries. And do you really need this transparency? But you should be very careful because implications could be big.
LIZ ALLEN: K, I think we have to stop there. Sorry everyone's slightly over, but I think there's lots to talk about. Thank you. [APPLAUSE]
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Twitter
Facebook
LinkedIn
Email
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode