Cadmore media player playing video Integrity by Design: Smart Approaches to Pilots & Partners
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
Integrity by Design: Smart Approaches to Pilots & Partners
Description:
Integrity by Design: Smart Approaches to Pilots & Partners
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/50e8d373-5fe0-4d31-acd8-83c6ca42ef33/thumbnails/50e8d373-5fe0-4d31-acd8-83c6ca42ef33.png
Duration:
T00H58M38S
Embed URL:
https://stream.cadmore.media/player/50e8d373-5fe0-4d31-acd8-83c6ca42ef33
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/50e8d373-5fe0-4d31-acd8-83c6ca42ef33/PSWS25 April - Research Integrity.mp4?sv=2019-02-02&sr=c&sig=p4zDtHJ9ODKIXuz%2FL6TIjBD8wxZCE4l2MUK%2BTayq%2Beg%3D&st=2025-04-24T13%3A22%3A41Z&se=2025-04-24T15%3A27%3A41Z&sp=r
Upload Date:
2025-04-21T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
STEPHANIE LOVEGROVE
HANSEN: Hello.
HANSEN: Welcome. My name is Stephanie Lovegrove Hansen. I'm the VP of marketing at Silverchair, and I want to thank you for joining us today. This is the second in the 2025 Platform Strategies Webinar Series, and I'm really looking forward to today's discussion. But first, I'm just going to go ahead and cover some logistics. So this webinar series features three virtual events. And you can find the registration links and recordings for all of those on the silverchair.com website.
HANSEN: Also be sure and save the date for our in-person Platform Strategies event, which will take place on September 25th in Washington, DC, and registration for that will open up next month. So this year's virtual series spotlights the biggest trends in publishing technology for 2025. And that includes accessibility, research integrity, which is what we'll be talking about today, and leveraging data to drive revenue.
HANSEN: So we designed these webinars intentionally to be more like a discussion among the panelists. And for that reason, we encourage you to participate. So you can do that via the chat or via the Q&A features throughout, and we will have some time at the end for Q&A. This event is being recorded. A copy of the recording will be made available freely on our website and be emailed to you afterwards. And finally, at the end of the event, you'll see a survey asking you for your feedback on the event, which just helps us to plan more of these in the future.
HANSEN: So thanks for offering your feedback. And with that, I'm very excited to have this group with us today. We've had some amazing discussions leading up to this, so I'm very much looking forward to today's conversation. We have with us Julia MacDonald, Director of Journals Product at Oxford University Press. Tiago Barros, co-founder of Signals. Adam Day, CEO and founder of clear skies.
HANSEN: And Joris van Rossum, product director at STM Integrity Hub. So before we I pass over to the panelists, I'm just going to do a little bit of scene setting. So the research integrity discussion, as we all know, has become increasingly urgent in recent years. And this is thanks in large part to the increasingly advanced methods for fraud and manipulation that are enabled by AI tools. The challenges also stem from incentives baked into the academic infrastructure, which demand fast and frequent publication from researchers.
HANSEN: And peer review, as we know, has long served as the first line of defense and serves as the hallmark of scholarly publishing. But as the scale of the research integrity challenges grow exponentially, we need the value of peer review to likewise scale. And so that's where pilots and partnerships come in, a lot of the things that we'll be discussing today. So in addition to the critical need for human peer reviewers, which is not going anywhere, publishers also need to leverage technology to assist them in filtering and flagging and technical checks.
HANSEN: And these kind of tools can be built in-house, can be bought off the shelf. But what we're really seeing more of is these partnerships with best in class service providers. These kind of partnerships have become more and more commonplace, but we wanted to convene today's panel to offer recommendations, insights from everyone's experience based on what they've seen in the partnerships that they've had and the things that they've tried over the course of the last few years.
HANSEN: So with that, I am going to let the speakers give a brief overview of their backgrounds, and then we'll get into our discussion. So we're going to start with Adam.
ADAM DAY: Hi. Let me just quickly share my screen. There we go. Yeah. So I'm Adam from Clear Skies. I think that the whole of research integrity sort of comes down to peer review in a way. Peer review is the place where science from one person meets other people and gets looked at.
ADAM DAY: And Clear Skies was actually founded based on that principle that we wanted to build tools that would support the peer review process. One thing that we did quite early on actually was journals that were using ScholarOne and compared with other journals. One thing that we found was that the alert rate that we see in publications from those journals is lower.
ADAM DAY: And I think one possible reason for that, and we never quite got right down to the bottom of it, was the existing tools already in ScholarOne, which are already helping publishers to deal with problematic papers. So Clear Skies uses a blend of artificial intelligence and network analysis to find signs of problematic behavior in a research manuscripts and also in published papers.
ADAM DAY: We've recently launched Oversight, which is our application which tracks and analyzes every single paper published in the last decade since 1st of January 2016. Clear Skies has won multiple awards for innovation, and we have not yet announced an integration with ScholarOne. However, we have already had publishers sign up. So I should probably say that there is integration available as of right now.
ADAM DAY: And when you use our integration, what you will see is an alert in ScholarOne and then full details of what we find in the Oversight platform. So thank you. We'll stop sharing now. STEPHANIE LOVEGROVE
HANSEN: All right.
HANSEN: Thanks, Adam. Joris, over to you.
JORIS VAN ROSSUM: Yes. Sorry about that. Joris van Rossum, product director, STM Integrity Hub, but also program director at STM Solutions. So STM Solutions is the operational arm of STM, and STM is the International Association of Scientific, Technical, and Medical Publishers. So about 3 and 1/2 years ago, publishers realized they needed to collaborate to address the challenges around research integrity, and at that time, we launched the Integrity Hub.
JORIS VAN ROSSUM: And the aim is to equip the scholarly communication community with data, intelligence, and technology to protect research integrity. We do that on three levels. Knowledge exchange between publishers on paper mills. We also develop frameworks and editorial policies. And last but not least, we built an enabling infrastructure, tools development and content pipelines to screen incoming manuscripts.
JORIS VAN ROSSUM: Our guiding principles. First of all, prevention instead of remediation, which means concretely that we want to catch issues at submission. So prevents those papers to enter the peer review process, let alone the published corpus. Collaboration with all stakeholders. That means publishers and STM members, non-STM members. But also submission, editorial systems, sleuth, et cetera.
JORIS VAN ROSSUM: Very important principle, we want to ensure that critical knowledge and tools are available to all publishers. We've seen in the last years that especially large publishers have built teams and technologies to screen manuscripts. You want to make sure that all the publishers of all shapes and sizes have access to those tools. Technology-driven was something that was mentioned at the beginning but always with the human in the loop.
JORIS VAN ROSSUM: Technology can support, but it's always the human being that makes a decision. And last but not the least, of course, fully upholding confidentiality and privacy standards. Something about where we are. After 3 and 1/2 years, we built two applications. One is an on-demand screening tool, allowing integrity specialists to upload manuscripts and screen them, but also ambient screening application, which means that submissions at publishers are pushed automatically at the hub.
JORIS VAN ROSSUM: For that, we have integrations with seven editorial systems, including ScholarOne, which means the content gets pushed to the hub, but also that we provide the signals in the environment of the editorial systems. We have now 15 tools integrated in both applications, and those include tools we build ourselves, but also third party tools, including Adam Day's tool. And we also launched a pilot program, so that our members can test tools, and once successful, we can roll them out to more publishers.
JORIS VAN ROSSUM: We have now 36 publishers using the applications. I should also mention that the hub is really the work of a lot of volunteers, from publishers and editorial systems as well. At the moment we screen more than 100,000 manuscripts. And according to our estimates, we identify dozens of paper mill manuscripts per day already. Thank you.
JORIS VAN ROSSUM: STEPHANIE LOVEGROVE
HANSEN: Great.
HANSEN: Thanks. Over to you, Tiago.
TIAGO BARROS: So I'm Tiago. I'm one of the co-founders of Signals. I'm really excited to be here today to tell you about Signals, what we do, and also in the context of Silverchair and the partnership that we have with ScholarOne. So a very quick overview of Signals, and I think it's interesting to go back to also what Stephanie said at the beginning about this issue around research integrity in the age of disinformation.
TIAGO BARROS: And really, that is at the core of Signal's mission. Our mission is around restoring trust in research. And that is, I would say that even from a personal point of view, that is very close to my heart because as a researcher of background and who has lots of friends and family that are researchers as well, that's hearing all of them talking about this issue, that they come across papers that they no longer know whether they can trust or not.
TIAGO BARROS: And that seems like a really fundamental problem that needs addressing. And that was also then one of the motivations to start Signals. What we do really is about analyzing the world's publication data and then combining that with expert knowledge. And specifically, we are developing this technology that we call the Signals data graph, in which we are combining that publication data that you can think of it as such as the articles and manuscripts themselves and their content, but also who the authors are, from which institutions they come from.
TIAGO BARROS: Also, looking at the citation network. And even, in many cases, with partnerships that we have with publishers, but also including proprietary publisher data that enhances the analysis. But we know that that may not be sufficient. And especially, because we know that there's lots of people out there with expertise in the fields that can contribute to the analysis that we can make for published content or manuscripts.
TIAGO BARROS: That is the second part of this Signals data graph around expert knowledge. And that comes in different flavors but three that I would like to highlight. One is the expert contributions from researchers themselves. That could be, for example, research integrity sleuths who make invaluable contributions to highlighting problems with papers out there, and we can get that data to inform our analysis.
TIAGO BARROS: But it can also be expertise from teams within the publishers themselves, whether that is their research integrity team, their editors, and also benefiting from the best practices that those publishers have to again inform our analysis. So taking all of this together, what we want to do is to provide the transparent and dynamic evaluation of the research output credibility. So that's really around linking back to the issue of research integrity in the age of disinformation.
TIAGO BARROS: And so as an example, there's a very quick overview of what a Signals evaluation of a publication looks like. And something that is central to us was this aspect of transparency. So it's not about just a black box saying that this paper has a problem and not exactly telling you why there is a problem with it. It really is about providing that transparent evaluation, which you as an editor or research integrity team member can look at at the scene of evaluation and very quickly see what is the problem that we identified with this publication, so that then you can make an informed decision on what to do with the publication itself.
TIAGO BARROS: And so those pieces of evidence that we can find, again, may relate to who the authors are and that track record, especially from a research integrity point of view. Or also, what is the foundations of this publication in terms of what other articles it's citing and whether there's problems already with those publications that may hint at this being also a problematic paper.
TIAGO BARROS: And of course, in the context of this webinar, I have to refer to the amazing collaboration that we've been doing with ScholarOne team and Silverchair. And one of the things that was really important for us is to go beyond this notion of a one time check for manuscripts as they come in because we think that it really maximize the value of what we can do when we look continuously at the entire editorial process or actually the entire publishing process of a manuscript.
TIAGO BARROS: So that every time there is new information that is available to us, that could, for example, could be linked to the peer review process itself as some of the other speakers were talking about before. So anytime there is new information, we can incorporate that information into our analysis and continuously update what Signals thinks about this manuscript itself.
TIAGO BARROS: And also just to say that this process doesn't even stop when the manuscript is published. Post-publication, there's continuous information that we can continue to acquire, again, for example, through the research integrity sleuth contributions that will continuously incorporate into the evaluation of the publication. And so they're therefore providing this very wide-ranging and comprehensive overview of the research integrity landscape.
TIAGO BARROS: STEPHANIE LOVEGROVE
HANSEN: Wonderful.
HANSEN: Thank you. And last but not least, Julia.
JULIA MCDONNELL: Well, I'll just get the slides up. Thank you. So I'm Julia. I am director of Journals Product. I lead the Journals Product team here at Oxford University Press or OUP, as I'm much more likely to call it from here on out. And I just wanted to start for a moment and say, a lot of people ask me, well, what on Earth does that mean? What does the Journals Product team do? And I like to summarize it as fundamentally, we do two main things.
JULIA MCDONNELL: We try to solve problems that are happening throughout the whole end to end journals publishing process, and we look to explore opportunities. And that involves lots of different things. But obviously, what we're going to talk about most here today is really the center option at the bottom there, where we're talking about pilots and projects. And in the context of trust and integrity, so one of the things that my team and I are responsible for is a dedicated trust and integrity program within OUP that sits alongside our journal ethics team, who are the people handling advice on complex ethics cases that arise.
JULIA MCDONNELL: And really, the program works very closely with that team. And together, we're trying to really drive a step change in terms of the processes and the level of confidence in the content that's being published. I think we all know that we move beyond a situation where you can default to an assumption of trust, and that takes a number of different forms. So very much, as Joris said, the more we can prevent this upstream, the better.
JULIA MCDONNELL: We don't want to be handling problems certainly post-publication. But also honestly, we don't really want to be handling them even at the point of submission. It's better if education and best practices and standards can catch things even earlier than that. But obviously, lots of problems can happen. And we need to have the capabilities to detect them, wherever that might happen in the process.
JULIA MCDONNELL: And that's obviously where a lot of the emphasis is in terms of pilots and looking at the sorts of technologies that are available in the market. And then I always like to say, it's not all bad news. It is also about signaling, which is really critical to help readers and users of content assess for themselves why they should trust this piece of content and why they should trust the authors of that content.
JULIA MCDONNELL: So really thinking about what we can do in that space to help drive that awareness, drive that transparency, and through that, help support informed trust when it comes to content. And I'm going to stop there. STEPHANIE LOVEGROVE
HANSEN: Great.
HANSEN: Well, thank you. That gives us a lot of good background to inform the discussion that we'll kick off now. So as we heard from everyone, many publishers are already engaged with a lot of these pilots and partnerships. There's a lot already going on. But at the same time, there's many more that are just now getting started in pursuing such offerings.
HANSEN: So what would you recommend from your experience that organizations should do in terms of how they approach addressing research integrity through partnerships? So for Julia, how does the build buy partner decision play out at OUP? And for our partners, what have you found are the key factors that play into partnership fit? And I'll let you kick that off, Julia.
JULIA MCDONNELL: Yeah. Thank you. So it's fairly obvious, but I think the first thing to say is you have to be really clear what you're trying to achieve in the first place. And that's critical to working through whatever your internal processes might look like to make that build buy partner decision. As part of that, it's really worth spending the time to think through what is your organization expert in?
JULIA MCDONNELL: What are you best placed to do yourself? Versus where can you tap into expertise beyond what you have available? I think when you get kind of past that point, you've sort of thought about it and gone, actually, we know we need to achieve something here, but we're not the best people to do it. We need to partner with somebody. Obviously, you've got to do your research.
JULIA MCDONNELL: And this is a very busy, active market because there's a lot changing, a lot that's evolving all the time. And it does really need that investment of time from whatever team is handling this within your organization, to go out, to talk to vendors, to attend the industry events, to really understand what is happening across the industry as a whole, to think through, well, which of the many different tools in any given space on integrity would be the best fit for you?
JULIA MCDONNELL: And that point about fit goes back again to understanding your goals in the first place. But it also ties into who are you as an organization? What is your mission? In the case of OUP, are you a not for profit? What are you what are you thinking about in those contexts? And how can you identify which potential partners might be best aligned with your strategic priorities and your ethos?
JULIA MCDONNELL: Obviously then, there's also the critical question of in a busy market, who do you have the greatest confidence in? Who has the credibility that this is going to deliver on all the promises that are being made and all the informational briefs. And link to that, and this will vary a lot depending on your organization. There's a question in there about what is your appetite for different levels of maturity?
JULIA MCDONNELL: Because it's moving so quickly, there are some brand new companies with brand new products that are very early in their product lifecycle, and that obviously requires a lot more time and knowledge from your side to be able to make that work effectively, versus increasingly, we're getting to a point where some of the capabilities have been around for a few years or in extensive use, and that's then a very different organizational lift to take part.
JULIA MCDONNELL: So yeah. Be really clear what you're trying to do, and that helps inform whether you should build, or buy, or partner. But if you are going to partner or buy, really think about that ethos and those priorities. STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: That's great. And I'm curious to hear from others, like you were saying, Julia, that the appetite for risk and the appetite for being the first one to try something. What have what have you all seen from your experience?
ADAM DAY: It's interesting that you mentioned risk because I feel like risk is the thing that comes up again and again in different contexts. One of the things that, when I was starting this, there wasn't anyone else doing any kind of paper mill detection or anything like that. So we had to figure out what were the right ways to do things responsibly. So we had things like in the publishing world.
ADAM DAY: We've got things like code guidelines, and we've got working groups like the ones that STM run, which are really great for working out what are the kind of ethical norms, and what are the rights and wrongs? But the other side was just looking at the legal side. Things like considering people's interests. Considering things like data protection, defamation, and IP copyright.
ADAM DAY: All of these things are different and intersect right at this kind of service. And so when you put these things together, the sort of confidentiality, the balancing of interests, and the need for purpose limitation and being really clear about exactly what is this data for, you get to need to know, it's really important to say, well, where does this data have to be? Who is it for?
ADAM DAY: Who owns it? And be very responsible about how that is handled. But I think that's a really key part of risk. It's really thinking things through at that level. And in that way, being prepared to handle that kind of volume of data.
JORIS VAN ROSSUM: Yeah. If I can add to that. I agree. I think about the kind of tools you would like to use, but think first about your own organization. Think about your challenges. So we're talking about paper mills. Some areas are more affected than others. So think about the challenges you had. Think about the specific issues you found.
JORIS VAN ROSSUM: But also think about your organization. Because the tools that we provide but others as well, they give a lot of signals at submission. So you need to make sure you have the teams. You have the expertise to handle those. That's what we saw in the first three years. Certainly, a lot of signals are coming to your organization. So think about how you want to handle that. Do you want to internal integrity teams to deal with those?
JORIS VAN ROSSUM: Do you want to give those signals directly to the editorial teams? Really think about those aspects as well.
TIAGO BARROS: Yeah. And if I can add. There's also a different perspective on risk, which is the risk of not doing things and what happens. And so actually, there's multiple ways to measure the risk. And I do think that given how much you can cancel the threat of research integrity issues that we see here are for many publishers, I think there's also a risk of being too paralyzed for whether we should do something or not.
TIAGO BARROS: But actually, something needs to be done in many cases. But then it becomes like more of a practical question of how do we manage that risk of the piloting and trying new things? And actually, I really like the fact that we started this discussion with links to peer review. Because in some ways, like the Signals team, half of it was the problems before. And we went through a similar experience in which, for example, at the time, there were these trends around open peer review and transparent peer review, and that was also seen by many publishers as a risk, like making that transition and something that was desirable but risky at the same time for the organization.
TIAGO BARROS: And so the kind of lessons that we learned from that experience that I think we're seeing again as being valuable in looking more into research integrity. And those are things like looking at the portfolio of a publisher and see what are the opportunities there. So it's not really the case that things have to be on and off all the time and across the entire portfolio of a publisher.
TIAGO BARROS: Maybe there are different teams within the publisher that have a different appetite for piloting. And so maybe leveraging those opportunities to learn something that then can be expanded across the organization. So that's practical steps that can mitigate the risk for the organization and to mitigate the risk of disruption of the operation. And by just being targeted, and again, as Julie said, with very clear, defined goals and objectives, so that then, with a small pilot, then can expand over time as the organization gains confidence in the partnership.
TIAGO BARROS: STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: And I completely agree with you about it seems like there's been a tipping point recently in terms of there's more risk in doing nothing. There's been enough high profile instances of research integrity issues at scale, that having things available like a pilot eliminates a lot of the risk while allowing you to address the underlying issue. And so let's say we've done the pilot. It's great. You're ready to move forward, expand it more broadly.
HANSEN: That's where a lot of challenges come in. How do you scale beyond the pilot? How does that transition present unique challenges as you move from a very specific use case to broader adoption? Curious to see what you've seen. And Adam, we can start with you.
ADAM DAY: Yeah. I mean, one of the things that we did quite early on was get straight up to industry scale. So the first time that we managed to process the whole of OpenAlex, that was April, 2023. So two years ago now. And so since then, we've been able to operate at industry scale. So we've been able to run processes at very high rates, which means we can adapt to things very quickly.
ADAM DAY: So as soon as we change our methods, we can just change our ratings on everything very quickly. And so in terms of scaling up with publishers, when publisher's data is coming in and that data goes up and up and up, it's fine. We're already processing data at that kind of scale. So the technical side is absolutely no problem.
ADAM DAY: The other side of it though is about how as more and more publishers get involved, the. diversity of what people need and what people are looking for has changed. And so that's been quite interesting because when this started, it was all about let's get the decision right as early as possible, right to the moment of submission. We want to get the alerts there, so people are forewarned as soon as something comes in.
ADAM DAY: I know I'm sort of echoing what Joris, what you said at the start. But it's really about getting things early as possible. So that decision get made. And the idea of that is to reduce the need to do investigations. It's to reduce the need to spend a lot of time and money on digging into all these different avenues that an investigation could go down.
ADAM DAY: And they can really go down rabbit holes. They can take a long, long time. And so really, we just wanted to avoid the whole need for that. As time has gone on, I think when I released the first version, the people who got involved straight away, were like, yes, we absolutely know what we want to do with that and straight away got on board. As it's developed over time and more people have come on board, it's more like oh, well, we want to use this for the early warnings.
ADAM DAY: But we also want to do investigations, and we want to figure out how to get those investigations as short as possible and really nip them in the bud. And so that diversity, that's been the interesting part, is seeing how as this has grown, the different kinds of use cases people have and I guess the differing expectations they have of the tools as they come aboard as well.
ADAM DAY: STEPHANIE LOVEGROVE
HANSEN: Anybody else
HANSEN: want to jump in on that?
TIAGO BARROS: Yeah. The scaling up also brings another challenge, which I think is quite interesting, which is there's a lot of unknown unknowns that you go into as you scale this pilot and implement us into a larger scale. We have discovered that we are experiencing already with some of our partnerships, the reality is that no two journals are the same, and there's often quite specific things on the operation, or just more generally, in the way that the journal operates that can actually have an impact on how things can scale from what the pilot was.
TIAGO BARROS: And I think that's something that we should accept and just be prepared that as things go up, often, it's not just a case of doing the same thing over and over again. But it's actually becoming slightly more complex and slightly more customized for specific use cases. And I actually find that very thrilling, if I have confessed because it feels like it never gets boring in some ways.
TIAGO BARROS: It's this excitement of looking at even sometimes, even within the same publisher but different, slightly different subject areas. They have different operations, slightly different standards on things that we look into. And so just seeing that evolution is quite interesting. So yeah. I agree with Adam. It's not really a technical limitation, which some often think, people think that it is.
TIAGO BARROS: But again, I always refer to, and I just think about every time anyone clicks I like in Facebook, that's a lot more data than what we deal with. And so that's not an issue. It's more about adapting to the specific use cases of the publisher.
JULIA MCDONNELL: Yeah. So I would say kind of from a publisher side of things, I agree with what you said, but I would come at it from a little bit of a different perspective. I think you need to be really clear on what is your target end state. So you've run a pilot, and you've gone great. This is useful for us. Which context is it useful for you in? What is the right scale?
JULIA MCDONNELL: Some of these tools are going to run on every piece of submitted content. Others are much more specialist, and you're only going to run them on certain articles. Maybe the ones that are focused on image manipulation or duplication. You're not going to run that on every paper. And so one of the things I think that's really important is when you're designing your pilot in the first place, is to make sure that pilot design process is going to give you as much as possible, in the face of the unknown unknowns, the answer to your questions about, well, what should this look like for us in a business as usual situation?
JULIA MCDONNELL: And there's also your really practical considerations here of no one's going to argue that we all need to take action here to reduce the risk, but these tools do add up to substantial additional costs, which may or may not be offset. You've kind of got to be in a BOU to see that. And particularly if you're maybe a smaller organization, there's going to be some really practical questions that you need to ask yourself about what is the balance that you can strike between tool used to help you mitigate risk and the cost, whether that's monetary or resource cost of managing these capabilities?
JULIA MCDONNELL: Particularly because I think it's fair to say very few publishers are piloting one thing. Most of us are piloting lots of things in parallel because there's so much happening here. So your end state for any given pilot is going to depend on the context of all the other things that you're doing in this space. And I think just the last thing I'll say on this, the other thing that's really important is when you are coming towards the end of a pilot, make sure you properly finish that pilot process and do a retrospective.
JULIA MCDONNELL: Because so many of the questions you're going to have to work through in terms of practically for your journals and your processes, how are you going to scale this up? Is it as simple as turn it on those journals and you're done? Well, probably not because even if that's the technical side of things, there's almost certainly an education and training element, and you're going to have questions that you need to be able to answer.
JULIA MCDONNELL: And you will have done a lot of that in the initial stage of the pilot. But it's how do you refine that in light of everything you've learned during the pilot to make it as smooth as possible to use this at a larger scale? So yeah. Really make sure that you design your pilots well would be my main thing to help you actually scale effectively. STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: And that's a great point about it's so much easier to identify what some of the challenges are or problem areas to key areas when it is in the pilot before you're dealing with it at the large scale. So that's a really good point to make sure you've closed that loop before moving on. Joris. I saw you unmuted. Do you have anything to add?
HANSEN: OK. All right. Well, one thing I really love about this panel today is that it represents a wide variety of backgrounds. We have publisher, editorial, researcher, service provider, product backgrounds. So from your unique perspectives, what would you say are the biggest roadblocks that you see, either to research integrity broadly or to partnerships specifically?
HANSEN: I know we had a great time in some of the planning calls, talking about this, and everyone has such interesting, unique perspectives that we'd love to hear from you all. So Joris, can you kick us off?
JORIS VAN ROSSUM: Yeah. One of the challenges that we see is that we're moving from an era of signal scarcity to one of signal abundance. So a few years ago, there were simply fewer tools that publishers could use when screening manuscripts. And now, they have access to many tools, which is, of course, a great development, but it again poses challenges in terms of resources and workflow. So our focus is on fine tuning and customizing these signals and targeting to specific people, specific moments within specific systems in the workflow.
JORIS VAN ROSSUM: Also, we don't believe there is or there will be a silver bullet signal. So it means that the value in integrity screening lies in aggregating different signals and bringing those to the right people in the right moment of the workflow. Maybe the largest challenge is GenAI because it threatens to change the way fraud is conducted. Our current screening system is based on copying or manipulating existing content.
JORIS VAN ROSSUM: So think about plagiarism, which is manipulating something existing. Image duplication, manipulating or changing something existing. And the whole screening paradigm is based on that feature. Of course, GenAI enables people to create text and images from scratch, which is far more challenging to detect. For this reason, alongside our efforts in the hub, STM is going what we call beyond the manuscript. So we believe in the short to mid-term, the tools that we talk about today are going to be very important, and they will remain very important.
JORIS VAN ROSSUM: But we need also to explore new ways to validate. For example, the identity of researchers, that's one thing we're focusing on. But also verifying the authenticity of images and other research data, which is equally important. I think there was a research a few years ago that 70% of all integrity breaches related to images. So that's going to be extremely important.
JORIS VAN ROSSUM: In terms of opportunity, maybe the last aspect. So once again, the biggest opportunity is collaboration. I believe we are all united in our efforts to protect research integrity. Institutions, editors, publishers, companies like those in the panel today, sleuth, submission systems, of course, and of course, also researchers themselves, each bringing specific perspectives. So we already demonstrated that collaboration is not only possible, but it's also leading to very concrete, tangible results.
JORIS VAN ROSSUM: STEPHANIE LOVEGROVE
HANSEN: Wonderful.
HANSEN: Yeah. Anyone else want to jump in? Biggest roadblock.
ADAM DAY: Joris said just about everything I could have thought of there.
STEPHANIE LOVEGROVE HANSEN: No other roadblocks.
JORIS VAN ROSSUM: I'm sorry.
JULIA MCDONNELL: I mean, I'm going to echo some of what Joris said. And I completely agree that's kind of one of the biggest challenges at the moment. Putting AI to one side, I talked about AI a lot yesterday, and you've already said it all, Joris. It's both a challenge and the opportunity, right? It is the proliferation of services. And the fact that at the moment, what we're seeing in that landscape is some true specialist services that are focused on very specific aspects.
JULIA MCDONNELL: But then increasingly, a range of services that are doing more than one thing that partially overlap with what then another service that's doing more than one thing is doing. And so it would be mad to try and run everything that's on the market on the same manuscript because you would just get so much noise. But I think we're sort of reaching hopefully the point where we're kind of at peak service proliferation, and we can start actually getting into a bit of a phase, less about consolidation, but more about integration.
JULIA MCDONNELL: As the announcements today have said, it's about having these system capabilities be part of your normal workflow in a way that the people who are making decisions and processing manuscripts. It's not something extra that they have to go find or do. It's just there. And where multiple services are being run, as much as possible, the outputs from those different results are being shown in a way that's really easy for you to interpret and assess across the different areas that have been looked at with those papers.
JULIA MCDONNELL: But I think of bigger picture, it wouldn't be a talk on trust and integrity if somebody didn't mention the systematic incentives that result in this content being created in the first place. And I think because those incentives are not likely to go away, we're going to continue to be stuck in this high demand for problematic behavior, whether that's individuals or whether that's through organizations and paper mills.
JULIA MCDONNELL: And they're really savvy. Paper mills in particular are really technologically savvy. And so we're in a bit of a Red Queen hypothesis situation, and it is an arms race. And everything that we do will then get countered, and we will have to develop and change the processes. And I'm sure Adam and Tiago can speak to what they see in terms of how quickly change is happening and how quickly we're having to adapt the tools.
JULIA MCDONNELL: So I think that's another one of the really big challenges that I don't really see going away, because that pressure is going to still be there.
ADAM DAY: Yeah. It's kind of funny what you say about the information overload part. So I used to edit journals and mostly using ScholarOne. And I went, I think in the time that I was doing that, it was something like 10,000 articles I went through. So it was a huge number of these. And the one thing I remember was that there was always more work than could be done in the time available. And so when creating these services, it was always about how do we minimize the amount of time that this is going to take up or reduce the amount of time that has already been taken up by other things?
ADAM DAY: And that was always a key goal. But there's something that comes from the data analysis part of my experience, which is I think it's a little bit counterintuitive, which is that usually, when you do data analysis, the more data you have, the better, right? The more examples you have. But not the more complicated your examples are. So if your examples have a lot of data in them, then what's interesting is that your ability to make predictions gets worse.
ADAM DAY: This is called the curse of dimensionality. And really, all it means is that like you say, if a human being is presented with a ton of information, it's very difficult to make a decision. It's actually no different for AI or for a piece of machine learning. The more data that comes in at once, the worse the outcome is. So when I started this, we had about 60 different ideas for things that we could look for in an article.
ADAM DAY: And I went through all of those. Some of them turned out to be terrible ideas. Some of them turned out to be things that worked. Some of them turned out to be things that worked. But then when you actually put them together, you realize that they were doing the same thing, and actually, you didn't need them. And we ended up with one out of those 60. Because that one was all that was needed at the start.
ADAM DAY: That's moved on from there, but it's always moved on with the same principle. It's about having the minimal amount of data. Because it's not just about the workload on the human who's using it. It's also actually about the quality of the analytics. The less diverse the data is that's going in, the better the outcome. And so yeah.
ADAM DAY: We're just about to release another version. And I think the total now is just up to about 20 different things. But it's not 20 different things that are right in your face. It's here are the important things. And the other things are there if you need to do the investigation and dig in. But ideally, we want to get to the decision as quickly as possible.
ADAM DAY: So they're right up top.
JORIS VAN ROSSUM: Yeah. In a way, it's the commoditization of signals. And having so many available also puts an emphasis on how we synthesize what those different signals are telling us. And so on one hand, again, providing the evidence, so that people can use it either for investigations or for action but also importantly on education. So we are aware of what is happening. And so we can educate the team and also the research community at large.
JORIS VAN ROSSUM: But also then the challenge of synthesizing that into something that is actionable and that people can make the right decision with the least amount of time possible, for sure. The other thing is also that we simplify things by talking about precision integrity, but it's actually that many different flavors of that. And to just think that all of that can be condensed to a single data point, it's wrong, right?
JORIS VAN ROSSUM: So this is the case of a data point, nor a binary thing quite often. So it's about thresholds. And it's like for risk and what is the acceptable norms in a specific context of the publication. So yeah. It is interesting to see how these things are coming all together.
JORIS VAN ROSSUM: And I think also, the other thing I want to say is there is quite often a focus on analyzing the content itself. But again, I think that leads exactly to what Julia was saying, that this arms race between we are getting better at spotting those things that we see in the content, but then bad actors are getting better at making our life very hard. That's definitely true. And it's also like why, the Signals, we always thought that it really cannot be just about the content.
JORIS VAN ROSSUM: It also has to be about the context of the publication and the incentives for people doing what they're doing, when you do it. And so in a way, if we can highlight the problems that are probably because of the incentives that are behind this malpractice, if we can address that and we can address the fact that maybe even if the content is not telling you very much, but you see what is the intention?
JORIS VAN ROSSUM: What is the source of that intention that we see? That already gives a lot of information that takes it away from the arms race or for ever more clever content. STEPHANIE LOVEGROVE
HANSEN: And that
HANSEN: brings me to another opportunity that we talked about in some of our planning, which is if these incentives exist that are causing a lot of these challenges, maybe we can't work that far upstream and address that. But what are the things we can do that do incentivize good practices? What are the ways that we can flag, and tag, and reward people or research that is coming through this in a positive way? What are the things we can do on our platforms and our tools, et cetera?
HANSEN: And it's interesting, Tiago, you talk a lot about, and I think throughout this conversation today, we're talking about technology partnerships, but a lot of what we've talked about is actually about people. The technology solution is only one part of the solution, but the people are really significant. So how do organizations define and maintain best practices for these specific areas? And that's going to be different obviously whether you're an OUP or a one journal society.
HANSEN: So how do you define those? Adam talked about cope a little bit. Like what are some of the best practices? How can we share with each other, learn from each other? And what role does education broadly play not only in addressing this as an industry but in these specific partnerships and pilots? Tiago, I will let you go first.
TIAGO BARROS: Yeah. That's a really good point. And it also goes back to one of the fundamentals I was talking about Signals in terms of this expert knowledge being added to our evaluations, which I think it really is at the core of this. So we can do lots of sophisticated technology that looks into networks and content and images and all of that. But there's still a huge opportunity to leverage the expert knowledge.
TIAGO BARROS: And again, that could be from research integrity sleuths, who are doing an amazing job highlighting problems that they come across. But it can also be like the combined knowledge within the publisher itself, that it shouldn't go to waste. It shouldn't just be focused around a single publication for which they spend time and apply their expertise to flex something and then just ends there. No.
TIAGO BARROS: It should be leveraged. It should be really utilized and really maximize the collective knowledge. So I think this is really important. Also the educational piece, these are different ways to look at it, I'll say. Again, there's the education in terms of understanding what are the incentives and what are the bad practices that we observe and to share that knowledge.
TIAGO BARROS: So everyone is aware. But perhaps it's not even about really education. It really is more about sharing of experiences and knowledge that I think if we can really leverage that more fully, I think it will really come through to the benefit of everyone. We all thought that there's a lot of focus on what publishers are doing around the research integrity issues. But by no means this is just a publisher's problem and they are the only stakeholders that need to do something about it.
TIAGO BARROS: So we have researchers that need to be involved. We have institutions that need to be involved. And so the share of knowledge and expertise and that combined efforts really is going to be what hopefully eventually will solve this problem. Again, the incentives are also there, not necessarily because of what publishers do, but because of the incentives that are leading to paper mills or beyond what publisher's action.
TIAGO BARROS: And so yeah. And the chair of knowledge also from the people, and again, going specifically into the topic of the pilots, that share of knowledge from the people that are involved in the initial pilot and then sharing the knowledge with the rest of the team within the publisher is really crucial. And again, so that's the extent, what the best ways to use the different tools or what to look for, how to give feedback to the vendors, so that they can continuously improve the tools.
TIAGO BARROS: I think that's really, really key.
JORIS VAN ROSSUM: Maybe very concretely. So with the help, we started with sharing knowledge. So we have various working groups. We have monthly meetings. So all the research integrity specialists, they come together. They share insights and experiences dealing with paper mills. How do you recognize, any new trends we see. We also have an image integrity working group, and they have created some promotional material.
JORIS VAN ROSSUM: So if you go to the STM website and look for the image integrity group. So we have also guidelines about how to detect image integrity. So I would say yeah, that sharing of knowledge amongst publishers, which is really, that was one of the big problems we had, right? Siloed nature of our way of working. And now, we created discussion groups and working groups to share that knowledge between publishers, but also encourage the sharing of information publishers with sleuth.
JORIS VAN ROSSUM: Every year, we have a Integrity Day in London in December, where we invite publishers, but also, for example, representatives from institutions and sleuth. When we started, we had about 70 people. Now, we have 250 last year. So you see also the community is getting larger, but also there's more interaction and engagement within that group.
JULIA MCDONNELL: I think if you only have-- completely agree with everything that's been said but taking it in a slightly different direction. If you only have resource to do one thing, you probably have to spend that on having some detection capabilities. But assuming you're big enough to do more than just that core basic work, I think it is really about making sure that you are thinking about this holistically because some of the best protection is, as Stephanie said, a lot of this is about people.
JULIA MCDONNELL: So some of the best protection is training your editors, is auditing your journal workflows and making sure that they are not-- that they don't have points of vulnerability in them that you're unaware of. And this goes to, as we know, paper mills are offering substantial monetary bribes. They are incentivized to corrupt the process. So there's a lot that you can do that isn't about technology.
JULIA MCDONNELL: That's about people and education and processes and policies to try and safeguard that process as much as possible. So very much, if you have the capability, make sure that you are not investing all of it just purely into technological detection because it's a lot bigger than that.
ADAM DAY: I think I have this memory from two or three years ago when people were still just sort of waking up to paper mills. It was sort of a period of time when I think a lot of publishers were aware, but there were some that still weren't. But especially people outside of publishing, you would go and talk to people about it and say, well, there's these people. They have these factories where they just make fake papers and sell them to people.
ADAM DAY: And you tell them that, and they just go, why on Earth would they do that? That's really weird. Because the incentives were not obvious to most people. And actually, they wouldn't have been obvious to me, going back maybe like a decade or so ago. And so I was editing journals, seeing papers all the time. I would still have found it really weird at that time that something like that was going on.
ADAM DAY: And it was going on at that time. And I think, yeah, the incentives for we're not going to solve it, are we? There's so much, it's so complex, and it's happening at such a different level to where, that we're never going to solve that. But I do think there is something in that education part of it.
ADAM DAY: Or maybe it's not education. Maybe it's just talking about stuff that we value, stuff that we care about. I think that really matters. It's like coming through like uni, I remember once getting a talk from a professor. And he was saying and explaining to people. Here's what's going to happen. If you ever get caught plagiarizing your work, you're going to get expelled immediately.
ADAM DAY: And I was like, I had never thought of that until you just said it. Because obviously, you only said that because that's what people do and the other people sitting in the room with me. But from my perspective, I think about it, I think the main things that would put me off doing something like that would be, number one, my values. Like I actually care about doing good science, and I care about doing things well.
ADAM DAY: I guess the other thing is just that I would find something like that to be immensely boring. The idea that you're just going to get a fake paper written and publish it. Like wouldn't it be more fun to just actually do some research? But yeah. I think that's it. It's so complex.
ADAM DAY: We can't figure it out how to solve that problem. But I do think there's a lot in just talking about what we care about. I think that really matters.
STEPHANIE LOVEGROVE HANSEN: I love the idea of fun is the positive incentive that we put for doing good research. Let's bring back fun. Love that. We've just got a couple minutes left. So I think one thing maybe to close on is we've talked about the service providers, the publishers. But Tiago, you touched on the fact that there's so many stakeholders involved in this conversation.
STEPHANIE LOVEGROVE HANSEN: There's the authors, there's the editors, there's the peer reviewers. That's a lot of stakeholder groups to work across. Any tips that you have for how to approach working across all of these groups as you approach these solutions?
TIAGO BARROS: Interesting question. So one thing that became very obvious to us is that the different stakeholders, although they have the shared responsibility and the research integrity problem, they all have very special needs and very need for specific products that are not going to be the same for everyone. So for example, again, I started the webinar by talking about my friends and family, the researchers that come to me and say, I come across this paper.
TIAGO BARROS: I don't know whether I can trust it or not. So having ways for researchers themselves to understand that and to know that whether you should trust or not to this paper I think becomes very important. But that's a very specific use case, right? That maybe the other stakeholders don't quite have exactly in the same way. And so I think that providing quite a bit product-centric view on these matters.
TIAGO BARROS: So I think that enabling creating products that meet specific needs from different stakeholders will, by itself, by their usage, by the sharing of the data that is used in the products, will bring together the stakeholders, even if they don't necessarily see that. So again, as I said, I talked about researchers also being involved. And that's why for example, when we bring closer together researchers to publishers in the sense that we leverage those comments from a peer or something that's research integrity has told Signals about, and we bring that to the eyes of a publisher, we're bringing them together, and so that we can leveraging that shared knowledge.
TIAGO BARROS: And the same thing goes for institutions. We're starting to see specific use cases where the same general principles that I use for detecting fraud or detecting fake submissions are hugely valuable for institutions when it comes to hiring, when it comes to promotion, and different use cases like this where the same principles can be applied. And again, that's a shared experience that we have with publishers, and researchers are feeding into those products that they're delivering to institutions.
TIAGO BARROS: STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: Any other quick thoughts before we wrap up? All right. Well, thank you so much. This has been a great conversation. Really appreciate all of you sharing your insights and excited to keep these conversations going. I feel like every conference these days has some research integrity topics from different angles. And I really enjoy hearing from everyone.
HANSEN: So thank you so much to our panelists today. Thank you to everyone who joined. Like I said, the recording will be made available online shortly, will be emailed to attendees. We also have one more event in the virtual series. I'll drop a quick link to the registration link. It is May 15th, and it's covering how to leverage your data across the publishing landscape. So thank you all again so much for joining today.
HANSEN: Really appreciate it. And hope everyone has a great rest of the day. Thank you.
ADAM DAY: Thank you, everyone.
Segment:0 .
STEPHANIE LOVEGROVE
HANSEN: Hello.
HANSEN: Welcome. My name is Stephanie Lovegrove Hansen. I'm the VP of marketing at Silverchair, and I want to thank you for joining us today. This is the second in the 2025 Platform Strategies Webinar Series, and I'm really looking forward to today's discussion. But first, I'm just going to go ahead and cover some logistics. So this webinar series features three virtual events. And you can find the registration links and recordings for all of those on the silverchair.com website.
HANSEN: Also be sure and save the date for our in-person Platform Strategies event, which will take place on September 25th in Washington, DC, and registration for that will open up next month. So this year's virtual series spotlights the biggest trends in publishing technology for 2025. And that includes accessibility, research integrity, which is what we'll be talking about today, and leveraging data to drive revenue.
HANSEN: So we designed these webinars intentionally to be more like a discussion among the panelists. And for that reason, we encourage you to participate. So you can do that via the chat or via the Q&A features throughout, and we will have some time at the end for Q&A. This event is being recorded. A copy of the recording will be made available freely on our website and be emailed to you afterwards. And finally, at the end of the event, you'll see a survey asking you for your feedback on the event, which just helps us to plan more of these in the future.
HANSEN: So thanks for offering your feedback. And with that, I'm very excited to have this group with us today. We've had some amazing discussions leading up to this, so I'm very much looking forward to today's conversation. We have with us Julia MacDonald, Director of Journals Product at Oxford University Press. Tiago Barros, co-founder of Signals. Adam Day, CEO and founder of clear skies.
HANSEN: And Joris van Rossum, product director at STM Integrity Hub. So before we I pass over to the panelists, I'm just going to do a little bit of scene setting. So the research integrity discussion, as we all know, has become increasingly urgent in recent years. And this is thanks in large part to the increasingly advanced methods for fraud and manipulation that are enabled by AI tools. The challenges also stem from incentives baked into the academic infrastructure, which demand fast and frequent publication from researchers.
HANSEN: And peer review, as we know, has long served as the first line of defense and serves as the hallmark of scholarly publishing. But as the scale of the research integrity challenges grow exponentially, we need the value of peer review to likewise scale. And so that's where pilots and partnerships come in, a lot of the things that we'll be discussing today. So in addition to the critical need for human peer reviewers, which is not going anywhere, publishers also need to leverage technology to assist them in filtering and flagging and technical checks.
HANSEN: And these kind of tools can be built in-house, can be bought off the shelf. But what we're really seeing more of is these partnerships with best in class service providers. These kind of partnerships have become more and more commonplace, but we wanted to convene today's panel to offer recommendations, insights from everyone's experience based on what they've seen in the partnerships that they've had and the things that they've tried over the course of the last few years.
HANSEN: So with that, I am going to let the speakers give a brief overview of their backgrounds, and then we'll get into our discussion. So we're going to start with Adam.
ADAM DAY: Hi. Let me just quickly share my screen. There we go. Yeah. So I'm Adam from Clear Skies. I think that the whole of research integrity sort of comes down to peer review in a way. Peer review is the place where science from one person meets other people and gets looked at.
ADAM DAY: And Clear Skies was actually founded based on that principle that we wanted to build tools that would support the peer review process. One thing that we did quite early on actually was journals that were using ScholarOne and compared with other journals. One thing that we found was that the alert rate that we see in publications from those journals is lower.
ADAM DAY: And I think one possible reason for that, and we never quite got right down to the bottom of it, was the existing tools already in ScholarOne, which are already helping publishers to deal with problematic papers. So Clear Skies uses a blend of artificial intelligence and network analysis to find signs of problematic behavior in a research manuscripts and also in published papers.
ADAM DAY: We've recently launched Oversight, which is our application which tracks and analyzes every single paper published in the last decade since 1st of January 2016. Clear Skies has won multiple awards for innovation, and we have not yet announced an integration with ScholarOne. However, we have already had publishers sign up. So I should probably say that there is integration available as of right now.
ADAM DAY: And when you use our integration, what you will see is an alert in ScholarOne and then full details of what we find in the Oversight platform. So thank you. We'll stop sharing now. STEPHANIE LOVEGROVE
HANSEN: All right.
HANSEN: Thanks, Adam. Joris, over to you.
JORIS VAN ROSSUM: Yes. Sorry about that. Joris van Rossum, product director, STM Integrity Hub, but also program director at STM Solutions. So STM Solutions is the operational arm of STM, and STM is the International Association of Scientific, Technical, and Medical Publishers. So about 3 and 1/2 years ago, publishers realized they needed to collaborate to address the challenges around research integrity, and at that time, we launched the Integrity Hub.
JORIS VAN ROSSUM: And the aim is to equip the scholarly communication community with data, intelligence, and technology to protect research integrity. We do that on three levels. Knowledge exchange between publishers on paper mills. We also develop frameworks and editorial policies. And last but not least, we built an enabling infrastructure, tools development and content pipelines to screen incoming manuscripts.
JORIS VAN ROSSUM: Our guiding principles. First of all, prevention instead of remediation, which means concretely that we want to catch issues at submission. So prevents those papers to enter the peer review process, let alone the published corpus. Collaboration with all stakeholders. That means publishers and STM members, non-STM members. But also submission, editorial systems, sleuth, et cetera.
JORIS VAN ROSSUM: Very important principle, we want to ensure that critical knowledge and tools are available to all publishers. We've seen in the last years that especially large publishers have built teams and technologies to screen manuscripts. You want to make sure that all the publishers of all shapes and sizes have access to those tools. Technology-driven was something that was mentioned at the beginning but always with the human in the loop.
JORIS VAN ROSSUM: Technology can support, but it's always the human being that makes a decision. And last but not the least, of course, fully upholding confidentiality and privacy standards. Something about where we are. After 3 and 1/2 years, we built two applications. One is an on-demand screening tool, allowing integrity specialists to upload manuscripts and screen them, but also ambient screening application, which means that submissions at publishers are pushed automatically at the hub.
JORIS VAN ROSSUM: For that, we have integrations with seven editorial systems, including ScholarOne, which means the content gets pushed to the hub, but also that we provide the signals in the environment of the editorial systems. We have now 15 tools integrated in both applications, and those include tools we build ourselves, but also third party tools, including Adam Day's tool. And we also launched a pilot program, so that our members can test tools, and once successful, we can roll them out to more publishers.
JORIS VAN ROSSUM: We have now 36 publishers using the applications. I should also mention that the hub is really the work of a lot of volunteers, from publishers and editorial systems as well. At the moment we screen more than 100,000 manuscripts. And according to our estimates, we identify dozens of paper mill manuscripts per day already. Thank you.
JORIS VAN ROSSUM: STEPHANIE LOVEGROVE
HANSEN: Great.
HANSEN: Thanks. Over to you, Tiago.
TIAGO BARROS: So I'm Tiago. I'm one of the co-founders of Signals. I'm really excited to be here today to tell you about Signals, what we do, and also in the context of Silverchair and the partnership that we have with ScholarOne. So a very quick overview of Signals, and I think it's interesting to go back to also what Stephanie said at the beginning about this issue around research integrity in the age of disinformation.
TIAGO BARROS: And really, that is at the core of Signal's mission. Our mission is around restoring trust in research. And that is, I would say that even from a personal point of view, that is very close to my heart because as a researcher of background and who has lots of friends and family that are researchers as well, that's hearing all of them talking about this issue, that they come across papers that they no longer know whether they can trust or not.
TIAGO BARROS: And that seems like a really fundamental problem that needs addressing. And that was also then one of the motivations to start Signals. What we do really is about analyzing the world's publication data and then combining that with expert knowledge. And specifically, we are developing this technology that we call the Signals data graph, in which we are combining that publication data that you can think of it as such as the articles and manuscripts themselves and their content, but also who the authors are, from which institutions they come from.
TIAGO BARROS: Also, looking at the citation network. And even, in many cases, with partnerships that we have with publishers, but also including proprietary publisher data that enhances the analysis. But we know that that may not be sufficient. And especially, because we know that there's lots of people out there with expertise in the fields that can contribute to the analysis that we can make for published content or manuscripts.
TIAGO BARROS: That is the second part of this Signals data graph around expert knowledge. And that comes in different flavors but three that I would like to highlight. One is the expert contributions from researchers themselves. That could be, for example, research integrity sleuths who make invaluable contributions to highlighting problems with papers out there, and we can get that data to inform our analysis.
TIAGO BARROS: But it can also be expertise from teams within the publishers themselves, whether that is their research integrity team, their editors, and also benefiting from the best practices that those publishers have to again inform our analysis. So taking all of this together, what we want to do is to provide the transparent and dynamic evaluation of the research output credibility. So that's really around linking back to the issue of research integrity in the age of disinformation.
TIAGO BARROS: And so as an example, there's a very quick overview of what a Signals evaluation of a publication looks like. And something that is central to us was this aspect of transparency. So it's not about just a black box saying that this paper has a problem and not exactly telling you why there is a problem with it. It really is about providing that transparent evaluation, which you as an editor or research integrity team member can look at at the scene of evaluation and very quickly see what is the problem that we identified with this publication, so that then you can make an informed decision on what to do with the publication itself.
TIAGO BARROS: And so those pieces of evidence that we can find, again, may relate to who the authors are and that track record, especially from a research integrity point of view. Or also, what is the foundations of this publication in terms of what other articles it's citing and whether there's problems already with those publications that may hint at this being also a problematic paper.
TIAGO BARROS: And of course, in the context of this webinar, I have to refer to the amazing collaboration that we've been doing with ScholarOne team and Silverchair. And one of the things that was really important for us is to go beyond this notion of a one time check for manuscripts as they come in because we think that it really maximize the value of what we can do when we look continuously at the entire editorial process or actually the entire publishing process of a manuscript.
TIAGO BARROS: So that every time there is new information that is available to us, that could, for example, could be linked to the peer review process itself as some of the other speakers were talking about before. So anytime there is new information, we can incorporate that information into our analysis and continuously update what Signals thinks about this manuscript itself.
TIAGO BARROS: And also just to say that this process doesn't even stop when the manuscript is published. Post-publication, there's continuous information that we can continue to acquire, again, for example, through the research integrity sleuth contributions that will continuously incorporate into the evaluation of the publication. And so they're therefore providing this very wide-ranging and comprehensive overview of the research integrity landscape.
TIAGO BARROS: STEPHANIE LOVEGROVE
HANSEN: Wonderful.
HANSEN: Thank you. And last but not least, Julia.
JULIA MCDONNELL: Well, I'll just get the slides up. Thank you. So I'm Julia. I am director of Journals Product. I lead the Journals Product team here at Oxford University Press or OUP, as I'm much more likely to call it from here on out. And I just wanted to start for a moment and say, a lot of people ask me, well, what on Earth does that mean? What does the Journals Product team do? And I like to summarize it as fundamentally, we do two main things.
JULIA MCDONNELL: We try to solve problems that are happening throughout the whole end to end journals publishing process, and we look to explore opportunities. And that involves lots of different things. But obviously, what we're going to talk about most here today is really the center option at the bottom there, where we're talking about pilots and projects. And in the context of trust and integrity, so one of the things that my team and I are responsible for is a dedicated trust and integrity program within OUP that sits alongside our journal ethics team, who are the people handling advice on complex ethics cases that arise.
JULIA MCDONNELL: And really, the program works very closely with that team. And together, we're trying to really drive a step change in terms of the processes and the level of confidence in the content that's being published. I think we all know that we move beyond a situation where you can default to an assumption of trust, and that takes a number of different forms. So very much, as Joris said, the more we can prevent this upstream, the better.
JULIA MCDONNELL: We don't want to be handling problems certainly post-publication. But also honestly, we don't really want to be handling them even at the point of submission. It's better if education and best practices and standards can catch things even earlier than that. But obviously, lots of problems can happen. And we need to have the capabilities to detect them, wherever that might happen in the process.
JULIA MCDONNELL: And that's obviously where a lot of the emphasis is in terms of pilots and looking at the sorts of technologies that are available in the market. And then I always like to say, it's not all bad news. It is also about signaling, which is really critical to help readers and users of content assess for themselves why they should trust this piece of content and why they should trust the authors of that content.
JULIA MCDONNELL: So really thinking about what we can do in that space to help drive that awareness, drive that transparency, and through that, help support informed trust when it comes to content. And I'm going to stop there. STEPHANIE LOVEGROVE
HANSEN: Great.
HANSEN: Well, thank you. That gives us a lot of good background to inform the discussion that we'll kick off now. So as we heard from everyone, many publishers are already engaged with a lot of these pilots and partnerships. There's a lot already going on. But at the same time, there's many more that are just now getting started in pursuing such offerings.
HANSEN: So what would you recommend from your experience that organizations should do in terms of how they approach addressing research integrity through partnerships? So for Julia, how does the build buy partner decision play out at OUP? And for our partners, what have you found are the key factors that play into partnership fit? And I'll let you kick that off, Julia.
JULIA MCDONNELL: Yeah. Thank you. So it's fairly obvious, but I think the first thing to say is you have to be really clear what you're trying to achieve in the first place. And that's critical to working through whatever your internal processes might look like to make that build buy partner decision. As part of that, it's really worth spending the time to think through what is your organization expert in?
JULIA MCDONNELL: What are you best placed to do yourself? Versus where can you tap into expertise beyond what you have available? I think when you get kind of past that point, you've sort of thought about it and gone, actually, we know we need to achieve something here, but we're not the best people to do it. We need to partner with somebody. Obviously, you've got to do your research.
JULIA MCDONNELL: And this is a very busy, active market because there's a lot changing, a lot that's evolving all the time. And it does really need that investment of time from whatever team is handling this within your organization, to go out, to talk to vendors, to attend the industry events, to really understand what is happening across the industry as a whole, to think through, well, which of the many different tools in any given space on integrity would be the best fit for you?
JULIA MCDONNELL: And that point about fit goes back again to understanding your goals in the first place. But it also ties into who are you as an organization? What is your mission? In the case of OUP, are you a not for profit? What are you what are you thinking about in those contexts? And how can you identify which potential partners might be best aligned with your strategic priorities and your ethos?
JULIA MCDONNELL: Obviously then, there's also the critical question of in a busy market, who do you have the greatest confidence in? Who has the credibility that this is going to deliver on all the promises that are being made and all the informational briefs. And link to that, and this will vary a lot depending on your organization. There's a question in there about what is your appetite for different levels of maturity?
JULIA MCDONNELL: Because it's moving so quickly, there are some brand new companies with brand new products that are very early in their product lifecycle, and that obviously requires a lot more time and knowledge from your side to be able to make that work effectively, versus increasingly, we're getting to a point where some of the capabilities have been around for a few years or in extensive use, and that's then a very different organizational lift to take part.
JULIA MCDONNELL: So yeah. Be really clear what you're trying to do, and that helps inform whether you should build, or buy, or partner. But if you are going to partner or buy, really think about that ethos and those priorities. STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: That's great. And I'm curious to hear from others, like you were saying, Julia, that the appetite for risk and the appetite for being the first one to try something. What have what have you all seen from your experience?
ADAM DAY: It's interesting that you mentioned risk because I feel like risk is the thing that comes up again and again in different contexts. One of the things that, when I was starting this, there wasn't anyone else doing any kind of paper mill detection or anything like that. So we had to figure out what were the right ways to do things responsibly. So we had things like in the publishing world.
ADAM DAY: We've got things like code guidelines, and we've got working groups like the ones that STM run, which are really great for working out what are the kind of ethical norms, and what are the rights and wrongs? But the other side was just looking at the legal side. Things like considering people's interests. Considering things like data protection, defamation, and IP copyright.
ADAM DAY: All of these things are different and intersect right at this kind of service. And so when you put these things together, the sort of confidentiality, the balancing of interests, and the need for purpose limitation and being really clear about exactly what is this data for, you get to need to know, it's really important to say, well, where does this data have to be? Who is it for?
ADAM DAY: Who owns it? And be very responsible about how that is handled. But I think that's a really key part of risk. It's really thinking things through at that level. And in that way, being prepared to handle that kind of volume of data.
JORIS VAN ROSSUM: Yeah. If I can add to that. I agree. I think about the kind of tools you would like to use, but think first about your own organization. Think about your challenges. So we're talking about paper mills. Some areas are more affected than others. So think about the challenges you had. Think about the specific issues you found.
JORIS VAN ROSSUM: But also think about your organization. Because the tools that we provide but others as well, they give a lot of signals at submission. So you need to make sure you have the teams. You have the expertise to handle those. That's what we saw in the first three years. Certainly, a lot of signals are coming to your organization. So think about how you want to handle that. Do you want to internal integrity teams to deal with those?
JORIS VAN ROSSUM: Do you want to give those signals directly to the editorial teams? Really think about those aspects as well.
TIAGO BARROS: Yeah. And if I can add. There's also a different perspective on risk, which is the risk of not doing things and what happens. And so actually, there's multiple ways to measure the risk. And I do think that given how much you can cancel the threat of research integrity issues that we see here are for many publishers, I think there's also a risk of being too paralyzed for whether we should do something or not.
TIAGO BARROS: But actually, something needs to be done in many cases. But then it becomes like more of a practical question of how do we manage that risk of the piloting and trying new things? And actually, I really like the fact that we started this discussion with links to peer review. Because in some ways, like the Signals team, half of it was the problems before. And we went through a similar experience in which, for example, at the time, there were these trends around open peer review and transparent peer review, and that was also seen by many publishers as a risk, like making that transition and something that was desirable but risky at the same time for the organization.
TIAGO BARROS: And so the kind of lessons that we learned from that experience that I think we're seeing again as being valuable in looking more into research integrity. And those are things like looking at the portfolio of a publisher and see what are the opportunities there. So it's not really the case that things have to be on and off all the time and across the entire portfolio of a publisher.
TIAGO BARROS: Maybe there are different teams within the publisher that have a different appetite for piloting. And so maybe leveraging those opportunities to learn something that then can be expanded across the organization. So that's practical steps that can mitigate the risk for the organization and to mitigate the risk of disruption of the operation. And by just being targeted, and again, as Julie said, with very clear, defined goals and objectives, so that then, with a small pilot, then can expand over time as the organization gains confidence in the partnership.
TIAGO BARROS: STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: And I completely agree with you about it seems like there's been a tipping point recently in terms of there's more risk in doing nothing. There's been enough high profile instances of research integrity issues at scale, that having things available like a pilot eliminates a lot of the risk while allowing you to address the underlying issue. And so let's say we've done the pilot. It's great. You're ready to move forward, expand it more broadly.
HANSEN: That's where a lot of challenges come in. How do you scale beyond the pilot? How does that transition present unique challenges as you move from a very specific use case to broader adoption? Curious to see what you've seen. And Adam, we can start with you.
ADAM DAY: Yeah. I mean, one of the things that we did quite early on was get straight up to industry scale. So the first time that we managed to process the whole of OpenAlex, that was April, 2023. So two years ago now. And so since then, we've been able to operate at industry scale. So we've been able to run processes at very high rates, which means we can adapt to things very quickly.
ADAM DAY: So as soon as we change our methods, we can just change our ratings on everything very quickly. And so in terms of scaling up with publishers, when publisher's data is coming in and that data goes up and up and up, it's fine. We're already processing data at that kind of scale. So the technical side is absolutely no problem.
ADAM DAY: The other side of it though is about how as more and more publishers get involved, the. diversity of what people need and what people are looking for has changed. And so that's been quite interesting because when this started, it was all about let's get the decision right as early as possible, right to the moment of submission. We want to get the alerts there, so people are forewarned as soon as something comes in.
ADAM DAY: I know I'm sort of echoing what Joris, what you said at the start. But it's really about getting things early as possible. So that decision get made. And the idea of that is to reduce the need to do investigations. It's to reduce the need to spend a lot of time and money on digging into all these different avenues that an investigation could go down.
ADAM DAY: And they can really go down rabbit holes. They can take a long, long time. And so really, we just wanted to avoid the whole need for that. As time has gone on, I think when I released the first version, the people who got involved straight away, were like, yes, we absolutely know what we want to do with that and straight away got on board. As it's developed over time and more people have come on board, it's more like oh, well, we want to use this for the early warnings.
ADAM DAY: But we also want to do investigations, and we want to figure out how to get those investigations as short as possible and really nip them in the bud. And so that diversity, that's been the interesting part, is seeing how as this has grown, the different kinds of use cases people have and I guess the differing expectations they have of the tools as they come aboard as well.
ADAM DAY: STEPHANIE LOVEGROVE
HANSEN: Anybody else
HANSEN: want to jump in on that?
TIAGO BARROS: Yeah. The scaling up also brings another challenge, which I think is quite interesting, which is there's a lot of unknown unknowns that you go into as you scale this pilot and implement us into a larger scale. We have discovered that we are experiencing already with some of our partnerships, the reality is that no two journals are the same, and there's often quite specific things on the operation, or just more generally, in the way that the journal operates that can actually have an impact on how things can scale from what the pilot was.
TIAGO BARROS: And I think that's something that we should accept and just be prepared that as things go up, often, it's not just a case of doing the same thing over and over again. But it's actually becoming slightly more complex and slightly more customized for specific use cases. And I actually find that very thrilling, if I have confessed because it feels like it never gets boring in some ways.
TIAGO BARROS: It's this excitement of looking at even sometimes, even within the same publisher but different, slightly different subject areas. They have different operations, slightly different standards on things that we look into. And so just seeing that evolution is quite interesting. So yeah. I agree with Adam. It's not really a technical limitation, which some often think, people think that it is.
TIAGO BARROS: But again, I always refer to, and I just think about every time anyone clicks I like in Facebook, that's a lot more data than what we deal with. And so that's not an issue. It's more about adapting to the specific use cases of the publisher.
JULIA MCDONNELL: Yeah. So I would say kind of from a publisher side of things, I agree with what you said, but I would come at it from a little bit of a different perspective. I think you need to be really clear on what is your target end state. So you've run a pilot, and you've gone great. This is useful for us. Which context is it useful for you in? What is the right scale?
JULIA MCDONNELL: Some of these tools are going to run on every piece of submitted content. Others are much more specialist, and you're only going to run them on certain articles. Maybe the ones that are focused on image manipulation or duplication. You're not going to run that on every paper. And so one of the things I think that's really important is when you're designing your pilot in the first place, is to make sure that pilot design process is going to give you as much as possible, in the face of the unknown unknowns, the answer to your questions about, well, what should this look like for us in a business as usual situation?
JULIA MCDONNELL: And there's also your really practical considerations here of no one's going to argue that we all need to take action here to reduce the risk, but these tools do add up to substantial additional costs, which may or may not be offset. You've kind of got to be in a BOU to see that. And particularly if you're maybe a smaller organization, there's going to be some really practical questions that you need to ask yourself about what is the balance that you can strike between tool used to help you mitigate risk and the cost, whether that's monetary or resource cost of managing these capabilities?
JULIA MCDONNELL: Particularly because I think it's fair to say very few publishers are piloting one thing. Most of us are piloting lots of things in parallel because there's so much happening here. So your end state for any given pilot is going to depend on the context of all the other things that you're doing in this space. And I think just the last thing I'll say on this, the other thing that's really important is when you are coming towards the end of a pilot, make sure you properly finish that pilot process and do a retrospective.
JULIA MCDONNELL: Because so many of the questions you're going to have to work through in terms of practically for your journals and your processes, how are you going to scale this up? Is it as simple as turn it on those journals and you're done? Well, probably not because even if that's the technical side of things, there's almost certainly an education and training element, and you're going to have questions that you need to be able to answer.
JULIA MCDONNELL: And you will have done a lot of that in the initial stage of the pilot. But it's how do you refine that in light of everything you've learned during the pilot to make it as smooth as possible to use this at a larger scale? So yeah. Really make sure that you design your pilots well would be my main thing to help you actually scale effectively. STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: And that's a great point about it's so much easier to identify what some of the challenges are or problem areas to key areas when it is in the pilot before you're dealing with it at the large scale. So that's a really good point to make sure you've closed that loop before moving on. Joris. I saw you unmuted. Do you have anything to add?
HANSEN: OK. All right. Well, one thing I really love about this panel today is that it represents a wide variety of backgrounds. We have publisher, editorial, researcher, service provider, product backgrounds. So from your unique perspectives, what would you say are the biggest roadblocks that you see, either to research integrity broadly or to partnerships specifically?
HANSEN: I know we had a great time in some of the planning calls, talking about this, and everyone has such interesting, unique perspectives that we'd love to hear from you all. So Joris, can you kick us off?
JORIS VAN ROSSUM: Yeah. One of the challenges that we see is that we're moving from an era of signal scarcity to one of signal abundance. So a few years ago, there were simply fewer tools that publishers could use when screening manuscripts. And now, they have access to many tools, which is, of course, a great development, but it again poses challenges in terms of resources and workflow. So our focus is on fine tuning and customizing these signals and targeting to specific people, specific moments within specific systems in the workflow.
JORIS VAN ROSSUM: Also, we don't believe there is or there will be a silver bullet signal. So it means that the value in integrity screening lies in aggregating different signals and bringing those to the right people in the right moment of the workflow. Maybe the largest challenge is GenAI because it threatens to change the way fraud is conducted. Our current screening system is based on copying or manipulating existing content.
JORIS VAN ROSSUM: So think about plagiarism, which is manipulating something existing. Image duplication, manipulating or changing something existing. And the whole screening paradigm is based on that feature. Of course, GenAI enables people to create text and images from scratch, which is far more challenging to detect. For this reason, alongside our efforts in the hub, STM is going what we call beyond the manuscript. So we believe in the short to mid-term, the tools that we talk about today are going to be very important, and they will remain very important.
JORIS VAN ROSSUM: But we need also to explore new ways to validate. For example, the identity of researchers, that's one thing we're focusing on. But also verifying the authenticity of images and other research data, which is equally important. I think there was a research a few years ago that 70% of all integrity breaches related to images. So that's going to be extremely important.
JORIS VAN ROSSUM: In terms of opportunity, maybe the last aspect. So once again, the biggest opportunity is collaboration. I believe we are all united in our efforts to protect research integrity. Institutions, editors, publishers, companies like those in the panel today, sleuth, submission systems, of course, and of course, also researchers themselves, each bringing specific perspectives. So we already demonstrated that collaboration is not only possible, but it's also leading to very concrete, tangible results.
JORIS VAN ROSSUM: STEPHANIE LOVEGROVE
HANSEN: Wonderful.
HANSEN: Yeah. Anyone else want to jump in? Biggest roadblock.
ADAM DAY: Joris said just about everything I could have thought of there.
STEPHANIE LOVEGROVE HANSEN: No other roadblocks.
JORIS VAN ROSSUM: I'm sorry.
JULIA MCDONNELL: I mean, I'm going to echo some of what Joris said. And I completely agree that's kind of one of the biggest challenges at the moment. Putting AI to one side, I talked about AI a lot yesterday, and you've already said it all, Joris. It's both a challenge and the opportunity, right? It is the proliferation of services. And the fact that at the moment, what we're seeing in that landscape is some true specialist services that are focused on very specific aspects.
JULIA MCDONNELL: But then increasingly, a range of services that are doing more than one thing that partially overlap with what then another service that's doing more than one thing is doing. And so it would be mad to try and run everything that's on the market on the same manuscript because you would just get so much noise. But I think we're sort of reaching hopefully the point where we're kind of at peak service proliferation, and we can start actually getting into a bit of a phase, less about consolidation, but more about integration.
JULIA MCDONNELL: As the announcements today have said, it's about having these system capabilities be part of your normal workflow in a way that the people who are making decisions and processing manuscripts. It's not something extra that they have to go find or do. It's just there. And where multiple services are being run, as much as possible, the outputs from those different results are being shown in a way that's really easy for you to interpret and assess across the different areas that have been looked at with those papers.
JULIA MCDONNELL: But I think of bigger picture, it wouldn't be a talk on trust and integrity if somebody didn't mention the systematic incentives that result in this content being created in the first place. And I think because those incentives are not likely to go away, we're going to continue to be stuck in this high demand for problematic behavior, whether that's individuals or whether that's through organizations and paper mills.
JULIA MCDONNELL: And they're really savvy. Paper mills in particular are really technologically savvy. And so we're in a bit of a Red Queen hypothesis situation, and it is an arms race. And everything that we do will then get countered, and we will have to develop and change the processes. And I'm sure Adam and Tiago can speak to what they see in terms of how quickly change is happening and how quickly we're having to adapt the tools.
JULIA MCDONNELL: So I think that's another one of the really big challenges that I don't really see going away, because that pressure is going to still be there.
ADAM DAY: Yeah. It's kind of funny what you say about the information overload part. So I used to edit journals and mostly using ScholarOne. And I went, I think in the time that I was doing that, it was something like 10,000 articles I went through. So it was a huge number of these. And the one thing I remember was that there was always more work than could be done in the time available. And so when creating these services, it was always about how do we minimize the amount of time that this is going to take up or reduce the amount of time that has already been taken up by other things?
ADAM DAY: And that was always a key goal. But there's something that comes from the data analysis part of my experience, which is I think it's a little bit counterintuitive, which is that usually, when you do data analysis, the more data you have, the better, right? The more examples you have. But not the more complicated your examples are. So if your examples have a lot of data in them, then what's interesting is that your ability to make predictions gets worse.
ADAM DAY: This is called the curse of dimensionality. And really, all it means is that like you say, if a human being is presented with a ton of information, it's very difficult to make a decision. It's actually no different for AI or for a piece of machine learning. The more data that comes in at once, the worse the outcome is. So when I started this, we had about 60 different ideas for things that we could look for in an article.
ADAM DAY: And I went through all of those. Some of them turned out to be terrible ideas. Some of them turned out to be things that worked. Some of them turned out to be things that worked. But then when you actually put them together, you realize that they were doing the same thing, and actually, you didn't need them. And we ended up with one out of those 60. Because that one was all that was needed at the start.
ADAM DAY: That's moved on from there, but it's always moved on with the same principle. It's about having the minimal amount of data. Because it's not just about the workload on the human who's using it. It's also actually about the quality of the analytics. The less diverse the data is that's going in, the better the outcome. And so yeah.
ADAM DAY: We're just about to release another version. And I think the total now is just up to about 20 different things. But it's not 20 different things that are right in your face. It's here are the important things. And the other things are there if you need to do the investigation and dig in. But ideally, we want to get to the decision as quickly as possible.
ADAM DAY: So they're right up top.
JORIS VAN ROSSUM: Yeah. In a way, it's the commoditization of signals. And having so many available also puts an emphasis on how we synthesize what those different signals are telling us. And so on one hand, again, providing the evidence, so that people can use it either for investigations or for action but also importantly on education. So we are aware of what is happening. And so we can educate the team and also the research community at large.
JORIS VAN ROSSUM: But also then the challenge of synthesizing that into something that is actionable and that people can make the right decision with the least amount of time possible, for sure. The other thing is also that we simplify things by talking about precision integrity, but it's actually that many different flavors of that. And to just think that all of that can be condensed to a single data point, it's wrong, right?
JORIS VAN ROSSUM: So this is the case of a data point, nor a binary thing quite often. So it's about thresholds. And it's like for risk and what is the acceptable norms in a specific context of the publication. So yeah. It is interesting to see how these things are coming all together.
JORIS VAN ROSSUM: And I think also, the other thing I want to say is there is quite often a focus on analyzing the content itself. But again, I think that leads exactly to what Julia was saying, that this arms race between we are getting better at spotting those things that we see in the content, but then bad actors are getting better at making our life very hard. That's definitely true. And it's also like why, the Signals, we always thought that it really cannot be just about the content.
JORIS VAN ROSSUM: It also has to be about the context of the publication and the incentives for people doing what they're doing, when you do it. And so in a way, if we can highlight the problems that are probably because of the incentives that are behind this malpractice, if we can address that and we can address the fact that maybe even if the content is not telling you very much, but you see what is the intention?
JORIS VAN ROSSUM: What is the source of that intention that we see? That already gives a lot of information that takes it away from the arms race or for ever more clever content. STEPHANIE LOVEGROVE
HANSEN: And that
HANSEN: brings me to another opportunity that we talked about in some of our planning, which is if these incentives exist that are causing a lot of these challenges, maybe we can't work that far upstream and address that. But what are the things we can do that do incentivize good practices? What are the ways that we can flag, and tag, and reward people or research that is coming through this in a positive way? What are the things we can do on our platforms and our tools, et cetera?
HANSEN: And it's interesting, Tiago, you talk a lot about, and I think throughout this conversation today, we're talking about technology partnerships, but a lot of what we've talked about is actually about people. The technology solution is only one part of the solution, but the people are really significant. So how do organizations define and maintain best practices for these specific areas? And that's going to be different obviously whether you're an OUP or a one journal society.
HANSEN: So how do you define those? Adam talked about cope a little bit. Like what are some of the best practices? How can we share with each other, learn from each other? And what role does education broadly play not only in addressing this as an industry but in these specific partnerships and pilots? Tiago, I will let you go first.
TIAGO BARROS: Yeah. That's a really good point. And it also goes back to one of the fundamentals I was talking about Signals in terms of this expert knowledge being added to our evaluations, which I think it really is at the core of this. So we can do lots of sophisticated technology that looks into networks and content and images and all of that. But there's still a huge opportunity to leverage the expert knowledge.
TIAGO BARROS: And again, that could be from research integrity sleuths, who are doing an amazing job highlighting problems that they come across. But it can also be like the combined knowledge within the publisher itself, that it shouldn't go to waste. It shouldn't just be focused around a single publication for which they spend time and apply their expertise to flex something and then just ends there. No.
TIAGO BARROS: It should be leveraged. It should be really utilized and really maximize the collective knowledge. So I think this is really important. Also the educational piece, these are different ways to look at it, I'll say. Again, there's the education in terms of understanding what are the incentives and what are the bad practices that we observe and to share that knowledge.
TIAGO BARROS: So everyone is aware. But perhaps it's not even about really education. It really is more about sharing of experiences and knowledge that I think if we can really leverage that more fully, I think it will really come through to the benefit of everyone. We all thought that there's a lot of focus on what publishers are doing around the research integrity issues. But by no means this is just a publisher's problem and they are the only stakeholders that need to do something about it.
TIAGO BARROS: So we have researchers that need to be involved. We have institutions that need to be involved. And so the share of knowledge and expertise and that combined efforts really is going to be what hopefully eventually will solve this problem. Again, the incentives are also there, not necessarily because of what publishers do, but because of the incentives that are leading to paper mills or beyond what publisher's action.
TIAGO BARROS: And so yeah. And the chair of knowledge also from the people, and again, going specifically into the topic of the pilots, that share of knowledge from the people that are involved in the initial pilot and then sharing the knowledge with the rest of the team within the publisher is really crucial. And again, so that's the extent, what the best ways to use the different tools or what to look for, how to give feedback to the vendors, so that they can continuously improve the tools.
TIAGO BARROS: I think that's really, really key.
JORIS VAN ROSSUM: Maybe very concretely. So with the help, we started with sharing knowledge. So we have various working groups. We have monthly meetings. So all the research integrity specialists, they come together. They share insights and experiences dealing with paper mills. How do you recognize, any new trends we see. We also have an image integrity working group, and they have created some promotional material.
JORIS VAN ROSSUM: So if you go to the STM website and look for the image integrity group. So we have also guidelines about how to detect image integrity. So I would say yeah, that sharing of knowledge amongst publishers, which is really, that was one of the big problems we had, right? Siloed nature of our way of working. And now, we created discussion groups and working groups to share that knowledge between publishers, but also encourage the sharing of information publishers with sleuth.
JORIS VAN ROSSUM: Every year, we have a Integrity Day in London in December, where we invite publishers, but also, for example, representatives from institutions and sleuth. When we started, we had about 70 people. Now, we have 250 last year. So you see also the community is getting larger, but also there's more interaction and engagement within that group.
JULIA MCDONNELL: I think if you only have-- completely agree with everything that's been said but taking it in a slightly different direction. If you only have resource to do one thing, you probably have to spend that on having some detection capabilities. But assuming you're big enough to do more than just that core basic work, I think it is really about making sure that you are thinking about this holistically because some of the best protection is, as Stephanie said, a lot of this is about people.
JULIA MCDONNELL: So some of the best protection is training your editors, is auditing your journal workflows and making sure that they are not-- that they don't have points of vulnerability in them that you're unaware of. And this goes to, as we know, paper mills are offering substantial monetary bribes. They are incentivized to corrupt the process. So there's a lot that you can do that isn't about technology.
JULIA MCDONNELL: That's about people and education and processes and policies to try and safeguard that process as much as possible. So very much, if you have the capability, make sure that you are not investing all of it just purely into technological detection because it's a lot bigger than that.
ADAM DAY: I think I have this memory from two or three years ago when people were still just sort of waking up to paper mills. It was sort of a period of time when I think a lot of publishers were aware, but there were some that still weren't. But especially people outside of publishing, you would go and talk to people about it and say, well, there's these people. They have these factories where they just make fake papers and sell them to people.
ADAM DAY: And you tell them that, and they just go, why on Earth would they do that? That's really weird. Because the incentives were not obvious to most people. And actually, they wouldn't have been obvious to me, going back maybe like a decade or so ago. And so I was editing journals, seeing papers all the time. I would still have found it really weird at that time that something like that was going on.
ADAM DAY: And it was going on at that time. And I think, yeah, the incentives for we're not going to solve it, are we? There's so much, it's so complex, and it's happening at such a different level to where, that we're never going to solve that. But I do think there is something in that education part of it.
ADAM DAY: Or maybe it's not education. Maybe it's just talking about stuff that we value, stuff that we care about. I think that really matters. It's like coming through like uni, I remember once getting a talk from a professor. And he was saying and explaining to people. Here's what's going to happen. If you ever get caught plagiarizing your work, you're going to get expelled immediately.
ADAM DAY: And I was like, I had never thought of that until you just said it. Because obviously, you only said that because that's what people do and the other people sitting in the room with me. But from my perspective, I think about it, I think the main things that would put me off doing something like that would be, number one, my values. Like I actually care about doing good science, and I care about doing things well.
ADAM DAY: I guess the other thing is just that I would find something like that to be immensely boring. The idea that you're just going to get a fake paper written and publish it. Like wouldn't it be more fun to just actually do some research? But yeah. I think that's it. It's so complex.
ADAM DAY: We can't figure it out how to solve that problem. But I do think there's a lot in just talking about what we care about. I think that really matters.
STEPHANIE LOVEGROVE HANSEN: I love the idea of fun is the positive incentive that we put for doing good research. Let's bring back fun. Love that. We've just got a couple minutes left. So I think one thing maybe to close on is we've talked about the service providers, the publishers. But Tiago, you touched on the fact that there's so many stakeholders involved in this conversation.
STEPHANIE LOVEGROVE HANSEN: There's the authors, there's the editors, there's the peer reviewers. That's a lot of stakeholder groups to work across. Any tips that you have for how to approach working across all of these groups as you approach these solutions?
TIAGO BARROS: Interesting question. So one thing that became very obvious to us is that the different stakeholders, although they have the shared responsibility and the research integrity problem, they all have very special needs and very need for specific products that are not going to be the same for everyone. So for example, again, I started the webinar by talking about my friends and family, the researchers that come to me and say, I come across this paper.
TIAGO BARROS: I don't know whether I can trust it or not. So having ways for researchers themselves to understand that and to know that whether you should trust or not to this paper I think becomes very important. But that's a very specific use case, right? That maybe the other stakeholders don't quite have exactly in the same way. And so I think that providing quite a bit product-centric view on these matters.
TIAGO BARROS: So I think that enabling creating products that meet specific needs from different stakeholders will, by itself, by their usage, by the sharing of the data that is used in the products, will bring together the stakeholders, even if they don't necessarily see that. So again, as I said, I talked about researchers also being involved. And that's why for example, when we bring closer together researchers to publishers in the sense that we leverage those comments from a peer or something that's research integrity has told Signals about, and we bring that to the eyes of a publisher, we're bringing them together, and so that we can leveraging that shared knowledge.
TIAGO BARROS: And the same thing goes for institutions. We're starting to see specific use cases where the same general principles that I use for detecting fraud or detecting fake submissions are hugely valuable for institutions when it comes to hiring, when it comes to promotion, and different use cases like this where the same principles can be applied. And again, that's a shared experience that we have with publishers, and researchers are feeding into those products that they're delivering to institutions.
TIAGO BARROS: STEPHANIE LOVEGROVE
HANSEN: Yeah.
HANSEN: Any other quick thoughts before we wrap up? All right. Well, thank you so much. This has been a great conversation. Really appreciate all of you sharing your insights and excited to keep these conversations going. I feel like every conference these days has some research integrity topics from different angles. And I really enjoy hearing from everyone.
HANSEN: So thank you so much to our panelists today. Thank you to everyone who joined. Like I said, the recording will be made available online shortly, will be emailed to attendees. We also have one more event in the virtual series. I'll drop a quick link to the registration link. It is May 15th, and it's covering how to leverage your data across the publishing landscape. So thank you all again so much for joining today.
HANSEN: Really appreciate it. And hope everyone has a great rest of the day. Thank you.
ADAM DAY: Thank you, everyone.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode