Cadmore media player playing video PS26: Beyond Integrity: why quality is the next frontier for the research record
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
PS26: Beyond Integrity: why quality is the next frontier for the research record
Description:
PS26: Beyond Integrity: why quality is the next frontier for the research record
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/188c2774-fe10-4e2b-8695-af6e6b575f75/thumbnails/188c2774-fe10-4e2b-8695-af6e6b575f75.png
Duration:
T00H47M42S
Embed URL:
https://stream.cadmore.media/player/188c2774-fe10-4e2b-8695-af6e6b575f75
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/188c2774-fe10-4e2b-8695-af6e6b575f75/PS26_Beyond Integrity.mp4?sv=2019-02-02&sr=c&sig=80VM2sNX9MobZPQb9hPfNLufKSAiK%2FFz4xgexAWjhPo%3D&st=2026-10-08T22%3A27%3A14Z&se=2026-10-09T00%3A32%3A14Z&sp=r
Upload Date:
2026-09-30T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
DAWN MELLEY: OK. I'm Dawn Melley, senior director of publishing operations for IEEE. I've been with IEEE 26 years in two days.
JONATHAN WOAHN: It's on her watch.
DAWN MELLEY: Yeah. It's a countdown. But I've been in scholarly publishing since the late '80s. So I'm in charge of everything from peer review, peer review support through to product delivery to the final destination. And sometimes, that's still a printer even.
MIRIAM MAUS: Hi, everybody. My name is Miriam Maus. I'm the chief publishing officer at Institute of Physics Publishing based in the UK. I started my publishing career not quite as early as Dawn, but I've worked in scholarly publishing since the late '90s, and I started at Blackwell and then worked at Wiley and then Institute of Physics Publishing since 2021.
DUSTIN SMITH: Great. So I'm Dustin Smith, co-founder and CEO of Hum. My team dresses me, and they're taking a picture right now. So this is really meta. But this ties into the broader conversation that we want to have about quality. But I was born in Detroit. My grandfather was a tool and die maker for Ford Motor for 35 years. Tool and die is basically like the machines that build the machines.
DUSTIN SMITH: To draw a very clunky parallel to what we're doing here is ultimately, like we're not building the research artifacts. We're building the machines that ultimately put those out there. So I'm waiting for the infringement suit from Ford Motor any day. But ultimately, this is my direct tie-in to that. So it comes from the family heritage. But we've been noodling this notion of integrity. And I'm 10 years into the industry and still feel quite new.
DUSTIN SMITH: And you go to a conference like STM, and it's almost like ohm integrity. And ultimately, like a lot of the tooling around integrity everybody's committed to. But if you solve integrity, like you solve what is canonically integrity, you suppress paper mills. You ultimately have author identities. You have that all sorted out.
DUSTIN SMITH: Ultimately, what is left in terms of what happens at the front door, at triage? So from what we've gathered, you're talking maybe 3% to 5% of manuscripts that might have those integrity concerns. But if you look at the data coming off of ScholarOne, desk reject rates, quite a bit higher. There's a yawning gap there. So I just want to start with Miriam. When we think of quality in scholarly publishing, what are we talking about?
MIRIAM MAUS: Yeah. Well, I'm just going to say the research integrity issue is nowhere near being solved. So I don't know we're talking about it.
DUSTIN SMITH: I mean, we solved it. It's done. We're good. At least for the 45 minutes.
MIRIAM MAUS: Excellent. So I think where we are at the moment, I mean, we've been asking questions about research integrity for a while. And the question there is is this content fraudulent? Is there a real issue with it? As you say, it is a small amount of content that is actually fraudulent, but a small share of an ever-growing corpus means it's ever more. So let's assume we've solved that.
MIRIAM MAUS: The question then is does this piece of content make a contribution to the scientific discourse? And I think there the debate is quite live because quality can be measured in so many different ways. There's a question for us in the editorial space around quality of service to authors. Not everybody publishes articles with groundbreaking research, but does everybody deserve a quality share of attention from a reviewer?
MIRIAM MAUS: Is one question. Or where do we say, OK, the bar is now so low that actually, we're overstraining the system? So I think there is a quality about the submission that comes into the editorial office. And where do we need to set the bar there in the context of constraints and overwhelmed system of reviewers?
DUSTIN SMITH: If we compare it to automotive, which is my heritage, you know a car is of sufficient quality because the steering wheel doesn't fly off. Or you can touch and feel, or it takes you from one place to another. Dawn, when you think about programmatic effectively running in an industrial grade or institutional publishing program, how do you think about quality at that level?
DAWN MELLEY: I think the basic quality process remains the same. And there's still going to be a human bias in it. We have hordes, lots of volunteer people involved in the process of judging the quality of every submission. And they can't leave their biases at the door necessarily. We count on them as experts in their fields of interest and that, but that only goes so far. So we're going to get it wrong some part of the time. But I hope a good amount of the time, we get it right.
DAWN MELLEY: And most of the articles that we publish are valuable in some sense and are of a high quality to others in those fields of interest. The IEEE is fairly broad in what we cover. So some of the fields of interest are very small communities. Some are very large. I think in the smaller communities, they get it right, but they all know each other.
DAWN MELLEY: And so there's some concern about the people that they might be running into later in their careers and when they need an article reviewed. So what am I saying? The human bias is always going to be there. We do the best we can. The fundamental process of quality review in peer review hasn't changed and won't change. But how can we assist them??
DAWN MELLEY: Are there things that we can take away that humans don't have to think about to allow them to do a better job on assessing the quality of each submission?
DUSTIN SMITH: Can you give an example?
DAWN MELLEY: Of which part of--
DUSTIN SMITH: Of all. No. What can you take off their plate, so they can only focus on top of license?
DAWN MELLEY: Sure. I mean, one of the things that the IEEE Publishing Strategic Planning Committee had been talking about a number of years ago now is the algorithms and everything within the articles that we publish. Can we have an AI agent that would go in there and validate all that stuff, so that the human doesn't have to do it? We're still working on that, and I think we're getting close in the technology space to being able to do that.
DAWN MELLEY: But that's something that would give them more time to focus on the overall assessment of the article and not worry about some of the mechanical details within.
DUSTIN SMITH: OK. We're leaving Jonathan hanging out to dry. I swear he has a critical role here. We'll do some math problems live on stage here. So one of the impetuses for this conversation is it was a conversation with a senior exec at a commercial publisher who admitted that their defect rate was somewhere in the single digit percentages. And if you think about industrial chemistry-- yes, the raised eyebrows.
DUSTIN SMITH: If you think about industrial chemistry or industrial process, you're talking about nines of reliability. And ultimately, if you're talking about 5% defect rate, you're talking about 5,000 parts per million, which are defective. Ultimately, like how do you think about that? Like you're coming into this even fresher than I am. Like how are you thinking about that in terms of the robot consumption and new pricing of quality?
JONATHAN WOAHN: Well, I think this question of quality is an interesting question because there are two ways that we can think about what that actually means. The first is in parts per million. And then the second is around this idea of demand. And I think the challenge part of what we're seeing is that from an AI perspective, it's hard to be able to distinguish between the two.
JONATHAN WOAHN: And so if we think about parts per million, that's the process of making sure that we have the right checks that are in place, the right peer review process, the right approval process. It's like making sure that all the boxes get checked along the way. But on the caliber side, it's this question of is this good research? Is this applicable research?
JONATHAN WOAHN: Is it relevant research? Is it content that actually like applies to what the AI is looking for? And if we look at it from those two different perspectives, in an AI workflow, it doesn't really, today, have a good way to distinguish between the two. And so if we conflate them and say, five parts per million or 5,000 out of a million, the question is, well, what are we actually talking about there?
JONATHAN WOAHN: Are we actually talking about the error rate, or are we actually talking about the relevancy of this content to what the AI or the user that is leveraging the AI is actually looking to do with it? And that, to me is like one of the big questions when it comes to quality that we're trying to help figure out how to answer some of those things.
DUSTIN SMITH: So take your work forward two or three years. Are you going to be a force for good in promoting greater quality within the content of an article?
JONATHAN WOAHN: I mean, I don't know if there's a wrong-- I mean, there is a wrong answer to that question, and I don't know that I can-- I mean, absolutely yes. Absolutely yes. And we are not alone in that fight. There are multiple layers to solving that problem.
DUSTIN SMITH: I mean, can you sell that? Like do the value proposition to Miriam, who has bought recently? So really, I'm curious. So take it forward and say like I've got a new way to price content effectively for robots. And this may or may not be true, but ultimately, the individual portions need to perhaps be more nutritious than the whole article. And that may or may not be true, but what's the pitch?
JONATHAN WOAHN: Well, I'm trying to think of analogies that exist in current state, and I think that one of the most immediate analogies that come to place is how to think about the role of a platform, like a media platform, like Netflix, where it's like there is a ton of content that's on Netflix. There's just some baseline standard of what exists there. But part of the value proposition that Netflix brings to the market is this idea of they understand user behavior because they can see it, because they see how people are interacting with all the content on their platform.
JONATHAN WOAHN: Therefore, they are doing two things around that. They are producing content that is in line with the user behaviors and user demand, and they are licensing content that is relevant that they haven't produced, but they know that their users are going to it. So really, in that sense, it's making the parts worth more than the whole comes down to the ability to deliver the right information at the right time to the right audience at the right price.
JONATHAN WOAHN: And so if we think about the agentic workflow from that perspective, like it's about understanding the researcher, it's about understanding the user and their goals and their context, and what do they have access to, and what should they have access to, and what do they want to have access to? And making sure that their agent is getting access to all of the information that it needs that fit all of those criteria.
JONATHAN WOAHN: And I think that's, yeah.
MIRIAM MAUS: I'm going to jump in. I think this is a really compelling value proposition for the downstream end of publishing. I think that really works. But we as publisher, we have to remember that we also serve the same users as authors. And there, the value proposition is very different. And I think this is where some of the, I hesitate to call it tension, but some of the considerations at the moment for publishers, like IOP Publishing and others in the room here, come up.
MIRIAM MAUS: Because when a researcher comes to us as an author, they have a very different expectation of the kind of quality they want, and the tolerance for, as you call it, defects, is extremely low. Things go don't go wrong very often. But if they do wrong, the author at the center of that is in trouble, and often publishers are in trouble too. So I think there is the consumption of the content is now changing so much, but the creation of the content is actually staying pretty steady in my experience.
MIRIAM MAUS: I mean, the expectations aren't changing hugely. I think the way content is created now is changing. But when authors come to us, to our journals, all those submissions that we're getting, they aren't that different from where they were five, 10 years ago. And I say that somebody who publishes, including in areas like machine learning, kind of slight innovation there.
MIRIAM MAUS: But fundamentally, an author comes to a publisher because they want a piece of research validated. So the question for us is then what do we have to change at that end of the process to make sure that the research is still useful and relevant when it comes to the other end, to the consumption end?
DUSTIN SMITH: Dawn, how do you see journals programs evolving over the next three to five years in response to some of the pressures? I mean, IEEE, computer science adjacent, and frankly, the most legible to AI agents and models and the production. How do you see reckoning with effectively the oversupply at the front door?
DAWN MELLEY: It's absolutely a scale issue right now, right? Because we all see increases in submissions, I would guess, in the range of 20% to 30% at this point, depending on who you are and what field you're in. But we'll adjust to that. We know how to adjust to that. We've been adjusting to scale for years. The human resources are in shorter supply. So we're looking for more tools to help scale. But we'll get there.
DAWN MELLEY: And, I would rather be a little slow right now and get the quality judgment right. I don't feel like the pressure to perform quickly and to scale up quickly is trumping the need to make sure that we are paying attention to quality. But, what I'm hearing in this conversation really is like, it's all back to discovery again and different ways of making our content discoverable. And I think that's an issue we've come to in this industry a number of times repeatedly as technologies advance.
DAWN MELLEY: So to me, the current AI rage is not scary. It's just another technology that we have to learn to work with and to take advantage of and to leverage as best we can. I think quality of content, the important thing for us is to really have those human resources able to do a good job in judging quality. And what's a quality article to you will be different from what is to me perhaps, and that's OK.
DAWN MELLEY: We won't get it all right just for you, but we'll get it right for somebody. And I think what Miriam was saying about the process hasn't really changed. The tools change, but the fundamental process hasn't changed. And will it with AI? It'll get maybe better, more efficient, faster, maybe more consistently repeatable.
DAWN MELLEY: But I don't know that the fundamental judgment of quality of the end product will change from our reliance on human experts.
MIRIAM MAUS: I think what will change is, and it will show us the stats about desk rejects going up. I think what will change is that if you're there any concerns about your manuscript, you will not get a review anymore. And that's changed. I mean, we used to review much more of what we received, but technology is really helping us make those judgment calls way earlier than we used to. And that is a good thing, by the way, because that means that there's more resource available to deal with the quality content.
DAWN MELLEY: Just speaking to that, it's interesting because the desk reject rate is definitely going up quickly but at the other end, the acceptance rate, final acceptance rate is creeping up a little bit. So it's telling me that the desk rejection is getting it right, and we're able to move forward with better content generally. And so we can accept more of the content that makes it through that initial screening.
DUSTIN SMITH: So if you carefully read all of the ScholarOne materials, including their white papers, 2026, 33% is the average for the platform. It's great. Put it by the bedside table, knocks you out. But we're seeing individual journals up as much as 80% or 90% or 100% in terms of submissions year over year, conferences. ICLR was up 3x year over year.
DUSTIN SMITH: So when you're talking about that magnitude, I mean, 30%, you could go a little bit slower or at a head or at a little bit of technology. But what happens when the hyperproduction is at that level, where you're talking 3 to 5x year over year? How fundamentally are you reconsidering what the enterprise looks like? Is that more than an incremental change?
DAWN MELLEY: Fortunately, for me, I'm not involved with conference peer review. Because it is, particularly in certain conferences, it is a major issue. And I've been hearing about that. And so it's interesting from afar watching what's going on there. And the solution seems to be we need more tools to AI-enabled, AI-based tools to handle this for us.
DAWN MELLEY: Or lately, we've been talking about in conferences the requirement to review at least five submissions if you're submitting one, right? And if you're submitting three, then you got to review 15. So they're doing things like that to try to get through the scale. I don't know what their ultimate answer will be, except probably, as Miriam was saying, even more stringent prescreening, and any excuse to get something out of the pipeline basically will be acted on.
MIRIAM MAUS: When I joined IOP Publishing 5 and 1/2 years ago, the two things that were put to me as absolutely fundamental to the publishing strategy I had to execute was get more submissions. We don't get enough of the submissions that are available.
DUSTIN SMITH: Or the old days.
MIRIAM MAUS: And be faster. So these two things were like, that was literally all we talked about for a while. That kind of why is everybody else's submissions growing faster than ours? And why is everybody else, at the time, the bone open access publishers were really setting the pace for fast peer review. We're now talking about, just like you said, is speed actually a good sign, or is it a worrying sign?
MIRIAM MAUS: Do we need to just signal a little bit more that quality processes take time? But maybe more importantly, we're now thinking about hurdles to put in the way of authors that have manuscripts that are not publishable and actually making sure that submissions, either it's more of a consideration to submit to a journal. That is a total turnaround from where it was just five years ago, and a lot of that is related to technology and AI.
DAWN MELLEY: I would back that up. Just IEEE has got an active corps of volunteers who run us on a daily basis. And there's an ad hoc dedicated to putting friction into the submission process. And I just said to them all the other day, I'm like, this is a change from what I've been doing for the past 30 years. Give me a minute. How do I figure out how to make it harder for authors, right?
DAWN MELLEY: Oh, hang on.
DUSTIN SMITH: That is quite the change. I mean, especially accepting PDFs and lightweighting the process and whatever else. I'm curious, it seems bad to say, the lower end of the quality spectrum. So still publishable, but things that are more focused on reproducibility.
DAWN MELLEY: Sound science.
DUSTIN SMITH: Sound science. Where is that threshold? And how do you treat it maybe in a journal's portfolio or at a sound science journal level, especially with the ability to hyperproduce? So you have these friction elements which aren't really discerning quality. They're just making it harder. Now, it's effectively free to create a research-shaped object. So how do you make it harder to do that?
DUSTIN SMITH: Where's the lower bound of what is likely publishable in the next three to five years in a sound science journal?
MIRIAM MAUS: It's a tricky question.
DAWN MELLEY: It is. It is a very tricky question. I mean, we try to be clear with each of our journals, and there are 250 of them now, on what their purpose is, what their scope is, what they're trying to do. And so there are a whole tranche of journals that are not sound science. Tehey're looking for novelty. They're looking for things that move the science forward.
DAWN MELLEY: But there are journals that are sound science? And we try to make that clear as well. Except a funny thing at IEEE, even our sound science, broad scope open access journal, the volunteers running the peer review process treat it like all the others. And so it really ends up being more than sound science. And so the lower bound is tough, I think, for us. Because it's tough for us just to do the sound science right to begin with.
MIRIAM MAUS: Totally echo that. There is this question about where is our lowest threshold? Is it really live one, a really interesting one? Because I mean, we're a society of publishers. Our mission is to expand the impact of physics and as a publisher, to give as many physicists the opportunity to share their research as we can. So that really speaks for the fact that if your content is publishable, we will publish it.
MIRIAM MAUS: But we haven't adjusted the process. You're absolutely right. It's the same process that you go through with a slightly different reviewer form. So then there is that debate of where does just OK stop and good start? And that threshold is it's really hard to institutionalize. I think individual editors will know it. They will know it when they look at it.
MIRIAM MAUS: But to do it at the scale that most sound science journals operate is really hard. And the other observation I would make about sound science is that the demand to publish in those journals is hugely increasing. These are the fastest growing submission. Trends we're seeing across our portfolio if it's a sound science. So there are people out there who want to publish because they need a publication.
MIRIAM MAUS: And the question for us is then, well, how much support can they expect for that?
DUSTIN SMITH: It's easier than ever to take a frontier, say fable class or Astra class model, apply it to open data sets basically legible to AI agents, and produce a paper end to end. You can say whether it's good or not. But ultimately, there's incremental advances in human understanding from applying that compute to data, and there's a structured output on the other side. So I'm curious where, maybe thinking IEEE, because it's more computer science relevant.
DUSTIN SMITH: If you're doing that, is that just for the provenance of preprints or GitHub or publishing by Twitter? And where's the bottom of where you're interested in?
DAWN MELLEY: Funny story. At IEEE, we have volunteers who are experts in AI machine learning, and they are the ones who are the most negative about using it in our publishing processes. So that tells me something. But, it doesn't. So it doesn't matter how you ask the question, my answer is going to be the same. We are having a really hard time finding that bottom cut off.
DUSTIN SMITH: And in some ways, it seems like intuition of people you respect, and domain experts is the best proxy for now. Absolutely. I mean, we count on the editors in chief, the editors that work under them and the reviewers to know their communities. Communities is a big word at IEEE, right? To know their communities well enough to be able to determine that this is an article that the community would like to read.
MIRIAM MAUS: Shall we ask Jonathan what he makes of those considerations? Because we, I think as clear, we agonize over what quality is, and we really go deep. And it's a philosophical question for many of us. When you hear us discussing that, what do you think?
JONATHAN WOAHN: I honestly end up having to say a lot of questions that like I just find personally curious about your process and how you guys are thinking about this. Because one of the very popular websites is archive.org. And this is where any AI researcher is going and just like-- I mean, they're getting flooded. And there's no approval process. It's just you go, and you upload it, and it's there.
JONATHAN WOAHN: But one of the things that's happening there is it's like the cycles are just starting to move so fast of like when the information gets out to when other people are wanting to get access to it. And so the question, that one of the things that I'm thinking about, as I'm hearing you guys talking about this is like if submissions are going up and you continue to trying to figure out where that floor is, but there's also this pressure of just like timelines, where it's like you can't have-- I mean, maybe in some domains, you can have really long feedback cycles where you can have multiple rounds going back and forth.
JONATHAN WOAHN: But as other adjacent fields of science continue to accelerate, how do you respond to that? Because by the time, if it gets published in a journal, the audience that might be most relevant for might have already moved on. And so that's like one of the questions that's going through my mind, is like how do you keep pace there?
DAWN MELLEY: So I can say that in the IEEE corpus, the fields that are moving the fastest tend to be the computer science based ones. And in those fields, it's more desirable to publish in a conference than a journal. I think for that very reason. Conferences are quick. And so that's where the pressure is in those fields. In journals, for me, and I think after 26 years of working with the volunteers, quality will always be the trump of everything else.
DAWN MELLEY: So if we have to slow down to ensure quality, then that's what we'll do.
JONATHAN WOAHN: So there's-- my former life, I was a lean expert. I worked in manufacturing plants. And so I very much came to-- I deployed a lot of manufacturing plants, the Toyota production system. And one of the, and please don't be offended by this, but one of the fallacies that a lot of the plants that we would go to is they would say time equates to quality. And the idea with the Toyota production system and lean is like, it's not about time, but it's about preventing errors.
JONATHAN WOAHN: And there's this concept that's called poka-yoke, which is like this is what a lot of the Detroit auto manufacturers adopted in following Toyota's rise was this idea of poka-yoke, where it's like instead of fixing things beforehand or afterwards, it's preventing mistakes from even being possible to be made in the beginning. And so it's like if you're installing a handle onto a car, if before, if there were five different ways that could get installed, like the poka-yoke process is to say, well, there's only one way this can get installed and to do it correctly.
JONATHAN WOAHN: And so with lean and in manufacturing, the idea is like it's not about time, but it's about trying to figure out how to manufacture errors out of the process. And so I guess that's one of the questions that I have in thinking through the submission processes is, as you guys were talking about injecting time into the process and injecting friction, there's not necessarily a direct correlation between quality.
JONATHAN WOAHN: So I'm just curious how you think how you think about that.
MIRIAM MAUS: I think this process of being lean and making sure that you frontload your QA is definitely in peer review. I mean, we break down the process from submission to acceptance like by the minute almost. So we really know how much time we spend at each stage and what we want the outcome to be. And this is where really technology, and tomorrow, Dustin and I will talk about Alchemist Review, and tools like that and others are really coming to the fore because they help us to basically support humans in getting it right the first time.
MIRIAM MAUS: So you have a system that does some of the heavy lifting, and then you reduce the amount of time that a person needs to look at it. And you do a load of that before anything even goes out to reviewers. So that when we get to the reviewers, we tell them don't have to worry about, checking references, formatting all that. Just tell us what is the contribution this piece of content makes to the field of, insert name of name of journal discipline.
DAWN MELLEY: Now if only we were making widgets, it would be great. No mistakes ever. But the slowdown comes in, Miriam is saying, we have all these tools. And so we're focusing the human resources on the things that they really need to focus on. But the issue there is, again, scale. And slowing down comes as a function of trying to scale. Because the human resources are limited.
DAWN MELLEY: So although we're getting their tasks more focused and they don't have to worry about as much as kind of peripheral stuff as they used to, there are only so many of them. So I can't take 10 reviewers and say, you still have to get through this many articles a day times 2. So the slowdown is in that backlog that has to work its way through the system. And we can't speed that up if the human resources are limited.
DAWN MELLEY: So where do we find the experts to scale that process? And that's where you would maintain or improve on the speed.
DUSTIN SMITH: Prepare your questions. I'll ask a couple more. But just so you have them in the can. And then Steph will be around with microphones. So both of you are highly reputable society publishers. But not everybody has that sort of community and reputation. Imagine there were publishing processes where you were so hard up for reviewers. Your median reviewer is mediocre, and your bottom quartile, reviewer is actually quite poor.
DUSTIN SMITH: And ultimately, that probably leaves you open to some sort of slop fraction making its way through. It's a polished research artifact. I'm curious, Jonathan, as you think about basically sizing and pricing portfolios of content, what is the potentially nefarious downstream effect of slop content, where a human direct to consumer might be able to say, this is slop, just sort of like seeing Claude slop in an email or slides.
DUSTIN SMITH: But robots ultimately are sort of like very fast consumers of pieces and parts.
JONATHAN WOAHN: Yeah. Well, what I love about both of your responses to my question here is the quality bar continues to be maintained at a certain level that reputation needs to continue to be intact. And so if you're engineering the waste out of the process at the time, really what you're doing is you're putting in a tremendous amount of effort upfront into the work that you're actually generating.
JONATHAN WOAHN: And a lot of that is what your institutions have done over the past, 20, 30, 40, 50 years, is building a reputation, building trust. And because in a Google world, your domain has gravitas. And so Google learned over time which sources were great, was able to build the classifiers to say for these types of questions, these are the authoritative sources for this. In an agentic world right now, literally, it's every agent out there is working off their own search algorithm.
JONATHAN WOAHN: They're working off their own prioritization. They're off their own classifiers, off their own scoring. And so what they are going to learn over time based on the human feedback is the pre-work that you are putting into the creation and the quality of the content will play itself out over time, and the agents will start to learn more broadly which sources are good and reputable sources based off of their human interaction and the behavior of how the people interact with it.
JONATHAN WOAHN: And if they start to request certain sources or certain journals or certain authors. But if it is the slop, then it's not going to get requested. It's not going to get it's not going to get picked up. And so you're not going to start to earn that gravitas. But in some ways, similar to Dawn's comment from earlier, it's like these are new tools that we just need to figure out how to make sure that that information gets communicated properly to the agents, so they can start making those determinations.
MIRIAM MAUS: And I think we have some agency here as publishers as well. We can be proactive. And there's a lot of discussion in the industry at the moment about trust markers and embedding trust markers. I'm a big fan of those initiatives, I have to say, because they win on several levels. They win on knowing whether you're a human or a robot or an AI agent, knowing the different categories of content.
MIRIAM MAUS: But they also win at the content production level because it gives authors, especially authors from less developed research community, a signal about, OK, this is what I can expect from this type of journal. And this is the kind of article that has gone through a certain process. So I think there is a win here for us working with technology and technology partners.
DUSTIN SMITH: Splendid. Do we have to give up some of the mix?
DAWN MELLEY: This is a lot for an introvert. Take my mic.
DUSTIN SMITH: Doing great, Dawn. Who we got? Hands up. Anybody? Bravery?
AUDIENCE: I found myself, as you all were discussing and thinking about the topic of quality, tracing back to COVID as one of the last kind of great pressure tests to quickly working through research and getting it out and making it accessible. So I think scholarly publishing learned very quickly how to operate at that speed and volume. And AI feels like another test of that, but something substantially more permanent that we're going to be working through.
AUDIENCE: So I'm just curious. I know you're not necessarily medical publishers. But what did we learn, maybe from the COVID era that's relevant to what we're working through and learning now with the adoption of AI and the increase in submissions, et cetera?
MIRIAM MAUS: Like I say, I'm not a medical publisher, so there may be other people in the room who have a better answer. But I think what we learned from COVID and the acceleration of research then is that if you get it right, the winds are great. If you get it wrong, the pitfalls are really serious. And I think this is what we should be taking into the AI era. It's so many opportunities, but the challenges are also real, and we need to be mindful of those.
MIRIAM MAUS: That doesn't mean we have to not pursue the opportunities. But there is something about responsibility here as well and understanding what is possible.
DAWN MELLEY: I also think that there's a resource issue just in successfully navigating through the change that AI is bringing in a technology sense. Because right now, I would say some of the AI tools we're using is making, they're making us less productive, right? We need to help them develop to the point where they make us more productive. And we also need to bring in different types of technology resources to help us navigate this era in the COVID sense.
DAWN MELLEY: IEEE kind of bucked the trend. We didn't see a huge increase in submissions through COVID. It happened afterwards.
DUSTIN SMITH: Yeah. I think COVID was 20% and then ChatGPT was 45. And then now, to the moon and beyond.
AUDIENCE: I have more of a comment. My name is Jasper Simons. I'm with the American Psychological Association. To gentlemen from the platforms, I find the analogy with manufacturing a terrible analogy in our industry. We're not trying to achieve a one standard and try to replicate it. So I really would love to hear more ideas about how you can support the novelty in science. Thank you.
DUSTIN SMITH: Support the novelty in science. I mean, I think quality in a manufacturing standard, much easier. You have stress and strain curves. You ultimately fatigue testing in industrial chemistry. You're taking retain samples. You're looking at adulteration rates. It's really collapsing to fewer dimensions. And I think everybody here would admit, like this is a messier enterprise of science.
DUSTIN SMITH: This is a collective of humans who are trying to validate our wisdom as a species. This is a big project that we're doing together. You can decompose, and we do, into things like rigor, novelty, and impact, and those are things which are increasingly machine capable activities that can assist humans basically do things like a first read and follow a set of procedures as if it were a human colleague.
DUSTIN SMITH: So that's a totally fundamentally different process than manufacturing. And it's kind of jokey that we're looking at parts per million here. Because that clearly fails. That's too simple of a standard. However, on the other side, to take a look and say these are my actual defect rates. Like this is our rate of fraud.
DUSTIN SMITH: Or ultimately, to what extent have you taken retained samples? There's a lot of trust in the community. How often do you take a retained sample of manuscripts or articles and do an end to end deep dive, where you have multiple experts spending a day apiece taking it apart into pieces to understand the quality of your processes? So I think there are things to learn. You obviously can't lift and shift directly from the manufacturing community.
DUSTIN SMITH:
JONATHAN WOAHN: To some degree, I agree on it is a terrible analogy. And one of the clients that I worked with in my previous life was an airline manufacturer or airplane manufacturer. They created airplanes. And literally every single airplane that came off that line is its own thing. And one of the things that we were able to do in working with them was creating checklists, being able to have a checklist of here are the gates, the dates and gates.
JONATHAN WOAHN: These are the things that we need to be able to pass through. And the analogy, as I think about this particular domain, and as platform providers, as part of what we can help provide is structure. And the reason that the structure becomes very relevant here is because without some kind of structure, everything becomes bespoke, and it becomes very-- it just becomes very slow to be able to get product out the door, to be able to get interoperability.
JONATHAN WOAHN: And so one of the ways, as a simple example in the age of AI, how we can think about this is think about what model context protocol or MCP has done for interaction between content and an agent. And so maybe we're not defining what the manufacturing process actually looks like, but being able to take a step back and say, what is the structure? What is the form? What is the way that communication can take place to help get the information out there, and to make sure that those trust markers get included and they get surface to the agents.
JONATHAN WOAHN: And so as I think about agentic access to scholarly content, that I've been operating against is there's kind of like five different layers to it. And it's like we need to be able to say, what are the trust markers of what is quality content? And you need to be able to tie those trust markers to the actual content itself. Those trust markers have to get delivered to the agents in order to be able to carry through who the author is or the provenance of that content.
JONATHAN WOAHN: The agents themselves have to be able to know how to interpret that content and how to interpret those trust markers. And finally, they need to know how to be able to give that reporting back. Like not just back to the platform, but back to the publisher, back to the agent, back to the authors to be able to complete that full cycle. And so part of the value that we can provide here is helping to think through that high level structure and make sure that it's actually enforced and actually deployed, so that the content that is being generated actually gets well represented in the way that you want it to within the environment where it's being consumed.
DUSTIN SMITH: Jessica Miles.
AUDIENCE: Jessica Miles, the informed frontier. I wanted to raise the point that quality and impact, I think as many of us in this room know, are not always immediately obvious during an article's publication or evaluation. Someone raised the Specter of COVID, and I myself was an editor of a microbiology journal during the COVID pandemic. And it was supremely challenging to understand what research was important or impactful while we're living through these novel conditions.
AUDIENCE: And so I wanted to ask if maybe this permissiveness that we've implicitly treated as a source of potential defects or problems with the research record is actually more of a feature than a bug. In some ways, we can guess what will move the field forward, or we might know more in some cases rather than others. But by letting maybe more in than we foresee being impactful, we allow the possibility that research, whose value might not have been recognized at the time of publication is ultimately added to the scientific record and can be used later.
AUDIENCE: So I would just throw that out there.
MIRIAM MAUS: Yeah. I mean, I entirely agree, and I think there are many, many examples in science and research where actually, something that didn't seem like much to start with then became a big thing. And that is definitely, I mean, as publishers and particularly as domain specific publishers, we have to create the space for that. And in fact, the members of our societies-- in physics, there's a big concern about blue sky research being sort of sidelined and everything has to be applied.
MIRIAM MAUS: That's a really important aspect of publishing that you make space for considerations and big, big ideas. So yeah. I entirely agree.
DAWN MELLEY: One of the things that's being talked about within IEEE is there's one camp. There's an editor in chief who's doing a research project right now to develop an AI reviewer. So the AI will actually do the review. And then others who say, but then what happens to novelty? So yeah. It's absolutely a challenge. And novelty always has been a challenge.
MIRIAM MAUS: I'm going to say something because I think it fits here, but I'm worried that I leave the stage and haven't said it. I think we really had this conversation very much on the publisher level. So we talked about publishers, but there is also a journal level. And I think we mustn't forget that there are different journals have different characteristics, different approaches, different criteria, different types of editors.
MIRIAM MAUS: And that's actually where kind of maybe this sort of the opportunity to create those spaces for discourse, that's where that sits in my mind.
DUSTIN SMITH: OK. One more, and then we're wrapping it up. I don't speak sign language, but I do speak Steph.
AUDIENCE: And it's not so much a comment or it's not so much a question as a comment. I'm Heather Kotula with Access Innovations. We're very proud to be a Silverchair Universe member. And we've addressed the scale issue for both peer review and conference proceedings for Plus, SPIE, and ASCO. And if you'd like to hear about it, I'd be glad to talk at the break.
DUSTIN SMITH: Straight up pitch. That's a strong move. All right. Yeah. Well, thank you, everybody. Is it 5 after the hour? OK. So this is break time. Do crosspollinate.
DUSTIN SMITH: Meet some folks. Get some refreshments. Be back seated by 5 after. Thanks so much. [APPLAUSE]
Segment:0 .
DAWN MELLEY: OK. I'm Dawn Melley, senior director of publishing operations for IEEE. I've been with IEEE 26 years in two days.
JONATHAN WOAHN: It's on her watch.
DAWN MELLEY: Yeah. It's a countdown. But I've been in scholarly publishing since the late '80s. So I'm in charge of everything from peer review, peer review support through to product delivery to the final destination. And sometimes, that's still a printer even.
MIRIAM MAUS: Hi, everybody. My name is Miriam Maus. I'm the chief publishing officer at Institute of Physics Publishing based in the UK. I started my publishing career not quite as early as Dawn, but I've worked in scholarly publishing since the late '90s, and I started at Blackwell and then worked at Wiley and then Institute of Physics Publishing since 2021.
DUSTIN SMITH: Great. So I'm Dustin Smith, co-founder and CEO of Hum. My team dresses me, and they're taking a picture right now. So this is really meta. But this ties into the broader conversation that we want to have about quality. But I was born in Detroit. My grandfather was a tool and die maker for Ford Motor for 35 years. Tool and die is basically like the machines that build the machines.
DUSTIN SMITH: To draw a very clunky parallel to what we're doing here is ultimately, like we're not building the research artifacts. We're building the machines that ultimately put those out there. So I'm waiting for the infringement suit from Ford Motor any day. But ultimately, this is my direct tie-in to that. So it comes from the family heritage. But we've been noodling this notion of integrity. And I'm 10 years into the industry and still feel quite new.
DUSTIN SMITH: And you go to a conference like STM, and it's almost like ohm integrity. And ultimately, like a lot of the tooling around integrity everybody's committed to. But if you solve integrity, like you solve what is canonically integrity, you suppress paper mills. You ultimately have author identities. You have that all sorted out.
DUSTIN SMITH: Ultimately, what is left in terms of what happens at the front door, at triage? So from what we've gathered, you're talking maybe 3% to 5% of manuscripts that might have those integrity concerns. But if you look at the data coming off of ScholarOne, desk reject rates, quite a bit higher. There's a yawning gap there. So I just want to start with Miriam. When we think of quality in scholarly publishing, what are we talking about?
MIRIAM MAUS: Yeah. Well, I'm just going to say the research integrity issue is nowhere near being solved. So I don't know we're talking about it.
DUSTIN SMITH: I mean, we solved it. It's done. We're good. At least for the 45 minutes.
MIRIAM MAUS: Excellent. So I think where we are at the moment, I mean, we've been asking questions about research integrity for a while. And the question there is is this content fraudulent? Is there a real issue with it? As you say, it is a small amount of content that is actually fraudulent, but a small share of an ever-growing corpus means it's ever more. So let's assume we've solved that.
MIRIAM MAUS: The question then is does this piece of content make a contribution to the scientific discourse? And I think there the debate is quite live because quality can be measured in so many different ways. There's a question for us in the editorial space around quality of service to authors. Not everybody publishes articles with groundbreaking research, but does everybody deserve a quality share of attention from a reviewer?
MIRIAM MAUS: Is one question. Or where do we say, OK, the bar is now so low that actually, we're overstraining the system? So I think there is a quality about the submission that comes into the editorial office. And where do we need to set the bar there in the context of constraints and overwhelmed system of reviewers?
DUSTIN SMITH: If we compare it to automotive, which is my heritage, you know a car is of sufficient quality because the steering wheel doesn't fly off. Or you can touch and feel, or it takes you from one place to another. Dawn, when you think about programmatic effectively running in an industrial grade or institutional publishing program, how do you think about quality at that level?
DAWN MELLEY: I think the basic quality process remains the same. And there's still going to be a human bias in it. We have hordes, lots of volunteer people involved in the process of judging the quality of every submission. And they can't leave their biases at the door necessarily. We count on them as experts in their fields of interest and that, but that only goes so far. So we're going to get it wrong some part of the time. But I hope a good amount of the time, we get it right.
DAWN MELLEY: And most of the articles that we publish are valuable in some sense and are of a high quality to others in those fields of interest. The IEEE is fairly broad in what we cover. So some of the fields of interest are very small communities. Some are very large. I think in the smaller communities, they get it right, but they all know each other.
DAWN MELLEY: And so there's some concern about the people that they might be running into later in their careers and when they need an article reviewed. So what am I saying? The human bias is always going to be there. We do the best we can. The fundamental process of quality review in peer review hasn't changed and won't change. But how can we assist them??
DAWN MELLEY: Are there things that we can take away that humans don't have to think about to allow them to do a better job on assessing the quality of each submission?
DUSTIN SMITH: Can you give an example?
DAWN MELLEY: Of which part of--
DUSTIN SMITH: Of all. No. What can you take off their plate, so they can only focus on top of license?
DAWN MELLEY: Sure. I mean, one of the things that the IEEE Publishing Strategic Planning Committee had been talking about a number of years ago now is the algorithms and everything within the articles that we publish. Can we have an AI agent that would go in there and validate all that stuff, so that the human doesn't have to do it? We're still working on that, and I think we're getting close in the technology space to being able to do that.
DAWN MELLEY: But that's something that would give them more time to focus on the overall assessment of the article and not worry about some of the mechanical details within.
DUSTIN SMITH: OK. We're leaving Jonathan hanging out to dry. I swear he has a critical role here. We'll do some math problems live on stage here. So one of the impetuses for this conversation is it was a conversation with a senior exec at a commercial publisher who admitted that their defect rate was somewhere in the single digit percentages. And if you think about industrial chemistry-- yes, the raised eyebrows.
DUSTIN SMITH: If you think about industrial chemistry or industrial process, you're talking about nines of reliability. And ultimately, if you're talking about 5% defect rate, you're talking about 5,000 parts per million, which are defective. Ultimately, like how do you think about that? Like you're coming into this even fresher than I am. Like how are you thinking about that in terms of the robot consumption and new pricing of quality?
JONATHAN WOAHN: Well, I think this question of quality is an interesting question because there are two ways that we can think about what that actually means. The first is in parts per million. And then the second is around this idea of demand. And I think the challenge part of what we're seeing is that from an AI perspective, it's hard to be able to distinguish between the two.
JONATHAN WOAHN: And so if we think about parts per million, that's the process of making sure that we have the right checks that are in place, the right peer review process, the right approval process. It's like making sure that all the boxes get checked along the way. But on the caliber side, it's this question of is this good research? Is this applicable research?
JONATHAN WOAHN: Is it relevant research? Is it content that actually like applies to what the AI is looking for? And if we look at it from those two different perspectives, in an AI workflow, it doesn't really, today, have a good way to distinguish between the two. And so if we conflate them and say, five parts per million or 5,000 out of a million, the question is, well, what are we actually talking about there?
JONATHAN WOAHN: Are we actually talking about the error rate, or are we actually talking about the relevancy of this content to what the AI or the user that is leveraging the AI is actually looking to do with it? And that, to me is like one of the big questions when it comes to quality that we're trying to help figure out how to answer some of those things.
DUSTIN SMITH: So take your work forward two or three years. Are you going to be a force for good in promoting greater quality within the content of an article?
JONATHAN WOAHN: I mean, I don't know if there's a wrong-- I mean, there is a wrong answer to that question, and I don't know that I can-- I mean, absolutely yes. Absolutely yes. And we are not alone in that fight. There are multiple layers to solving that problem.
DUSTIN SMITH: I mean, can you sell that? Like do the value proposition to Miriam, who has bought recently? So really, I'm curious. So take it forward and say like I've got a new way to price content effectively for robots. And this may or may not be true, but ultimately, the individual portions need to perhaps be more nutritious than the whole article. And that may or may not be true, but what's the pitch?
JONATHAN WOAHN: Well, I'm trying to think of analogies that exist in current state, and I think that one of the most immediate analogies that come to place is how to think about the role of a platform, like a media platform, like Netflix, where it's like there is a ton of content that's on Netflix. There's just some baseline standard of what exists there. But part of the value proposition that Netflix brings to the market is this idea of they understand user behavior because they can see it, because they see how people are interacting with all the content on their platform.
JONATHAN WOAHN: Therefore, they are doing two things around that. They are producing content that is in line with the user behaviors and user demand, and they are licensing content that is relevant that they haven't produced, but they know that their users are going to it. So really, in that sense, it's making the parts worth more than the whole comes down to the ability to deliver the right information at the right time to the right audience at the right price.
JONATHAN WOAHN: And so if we think about the agentic workflow from that perspective, like it's about understanding the researcher, it's about understanding the user and their goals and their context, and what do they have access to, and what should they have access to, and what do they want to have access to? And making sure that their agent is getting access to all of the information that it needs that fit all of those criteria.
JONATHAN WOAHN: And I think that's, yeah.
MIRIAM MAUS: I'm going to jump in. I think this is a really compelling value proposition for the downstream end of publishing. I think that really works. But we as publisher, we have to remember that we also serve the same users as authors. And there, the value proposition is very different. And I think this is where some of the, I hesitate to call it tension, but some of the considerations at the moment for publishers, like IOP Publishing and others in the room here, come up.
MIRIAM MAUS: Because when a researcher comes to us as an author, they have a very different expectation of the kind of quality they want, and the tolerance for, as you call it, defects, is extremely low. Things go don't go wrong very often. But if they do wrong, the author at the center of that is in trouble, and often publishers are in trouble too. So I think there is the consumption of the content is now changing so much, but the creation of the content is actually staying pretty steady in my experience.
MIRIAM MAUS: I mean, the expectations aren't changing hugely. I think the way content is created now is changing. But when authors come to us, to our journals, all those submissions that we're getting, they aren't that different from where they were five, 10 years ago. And I say that somebody who publishes, including in areas like machine learning, kind of slight innovation there.
MIRIAM MAUS: But fundamentally, an author comes to a publisher because they want a piece of research validated. So the question for us is then what do we have to change at that end of the process to make sure that the research is still useful and relevant when it comes to the other end, to the consumption end?
DUSTIN SMITH: Dawn, how do you see journals programs evolving over the next three to five years in response to some of the pressures? I mean, IEEE, computer science adjacent, and frankly, the most legible to AI agents and models and the production. How do you see reckoning with effectively the oversupply at the front door?
DAWN MELLEY: It's absolutely a scale issue right now, right? Because we all see increases in submissions, I would guess, in the range of 20% to 30% at this point, depending on who you are and what field you're in. But we'll adjust to that. We know how to adjust to that. We've been adjusting to scale for years. The human resources are in shorter supply. So we're looking for more tools to help scale. But we'll get there.
DAWN MELLEY: And, I would rather be a little slow right now and get the quality judgment right. I don't feel like the pressure to perform quickly and to scale up quickly is trumping the need to make sure that we are paying attention to quality. But, what I'm hearing in this conversation really is like, it's all back to discovery again and different ways of making our content discoverable. And I think that's an issue we've come to in this industry a number of times repeatedly as technologies advance.
DAWN MELLEY: So to me, the current AI rage is not scary. It's just another technology that we have to learn to work with and to take advantage of and to leverage as best we can. I think quality of content, the important thing for us is to really have those human resources able to do a good job in judging quality. And what's a quality article to you will be different from what is to me perhaps, and that's OK.
DAWN MELLEY: We won't get it all right just for you, but we'll get it right for somebody. And I think what Miriam was saying about the process hasn't really changed. The tools change, but the fundamental process hasn't changed. And will it with AI? It'll get maybe better, more efficient, faster, maybe more consistently repeatable.
DAWN MELLEY: But I don't know that the fundamental judgment of quality of the end product will change from our reliance on human experts.
MIRIAM MAUS: I think what will change is, and it will show us the stats about desk rejects going up. I think what will change is that if you're there any concerns about your manuscript, you will not get a review anymore. And that's changed. I mean, we used to review much more of what we received, but technology is really helping us make those judgment calls way earlier than we used to. And that is a good thing, by the way, because that means that there's more resource available to deal with the quality content.
DAWN MELLEY: Just speaking to that, it's interesting because the desk reject rate is definitely going up quickly but at the other end, the acceptance rate, final acceptance rate is creeping up a little bit. So it's telling me that the desk rejection is getting it right, and we're able to move forward with better content generally. And so we can accept more of the content that makes it through that initial screening.
DUSTIN SMITH: So if you carefully read all of the ScholarOne materials, including their white papers, 2026, 33% is the average for the platform. It's great. Put it by the bedside table, knocks you out. But we're seeing individual journals up as much as 80% or 90% or 100% in terms of submissions year over year, conferences. ICLR was up 3x year over year.
DUSTIN SMITH: So when you're talking about that magnitude, I mean, 30%, you could go a little bit slower or at a head or at a little bit of technology. But what happens when the hyperproduction is at that level, where you're talking 3 to 5x year over year? How fundamentally are you reconsidering what the enterprise looks like? Is that more than an incremental change?
DAWN MELLEY: Fortunately, for me, I'm not involved with conference peer review. Because it is, particularly in certain conferences, it is a major issue. And I've been hearing about that. And so it's interesting from afar watching what's going on there. And the solution seems to be we need more tools to AI-enabled, AI-based tools to handle this for us.
DAWN MELLEY: Or lately, we've been talking about in conferences the requirement to review at least five submissions if you're submitting one, right? And if you're submitting three, then you got to review 15. So they're doing things like that to try to get through the scale. I don't know what their ultimate answer will be, except probably, as Miriam was saying, even more stringent prescreening, and any excuse to get something out of the pipeline basically will be acted on.
MIRIAM MAUS: When I joined IOP Publishing 5 and 1/2 years ago, the two things that were put to me as absolutely fundamental to the publishing strategy I had to execute was get more submissions. We don't get enough of the submissions that are available.
DUSTIN SMITH: Or the old days.
MIRIAM MAUS: And be faster. So these two things were like, that was literally all we talked about for a while. That kind of why is everybody else's submissions growing faster than ours? And why is everybody else, at the time, the bone open access publishers were really setting the pace for fast peer review. We're now talking about, just like you said, is speed actually a good sign, or is it a worrying sign?
MIRIAM MAUS: Do we need to just signal a little bit more that quality processes take time? But maybe more importantly, we're now thinking about hurdles to put in the way of authors that have manuscripts that are not publishable and actually making sure that submissions, either it's more of a consideration to submit to a journal. That is a total turnaround from where it was just five years ago, and a lot of that is related to technology and AI.
DAWN MELLEY: I would back that up. Just IEEE has got an active corps of volunteers who run us on a daily basis. And there's an ad hoc dedicated to putting friction into the submission process. And I just said to them all the other day, I'm like, this is a change from what I've been doing for the past 30 years. Give me a minute. How do I figure out how to make it harder for authors, right?
DAWN MELLEY: Oh, hang on.
DUSTIN SMITH: That is quite the change. I mean, especially accepting PDFs and lightweighting the process and whatever else. I'm curious, it seems bad to say, the lower end of the quality spectrum. So still publishable, but things that are more focused on reproducibility.
DAWN MELLEY: Sound science.
DUSTIN SMITH: Sound science. Where is that threshold? And how do you treat it maybe in a journal's portfolio or at a sound science journal level, especially with the ability to hyperproduce? So you have these friction elements which aren't really discerning quality. They're just making it harder. Now, it's effectively free to create a research-shaped object. So how do you make it harder to do that?
DUSTIN SMITH: Where's the lower bound of what is likely publishable in the next three to five years in a sound science journal?
MIRIAM MAUS: It's a tricky question.
DAWN MELLEY: It is. It is a very tricky question. I mean, we try to be clear with each of our journals, and there are 250 of them now, on what their purpose is, what their scope is, what they're trying to do. And so there are a whole tranche of journals that are not sound science. Tehey're looking for novelty. They're looking for things that move the science forward.
DAWN MELLEY: But there are journals that are sound science? And we try to make that clear as well. Except a funny thing at IEEE, even our sound science, broad scope open access journal, the volunteers running the peer review process treat it like all the others. And so it really ends up being more than sound science. And so the lower bound is tough, I think, for us. Because it's tough for us just to do the sound science right to begin with.
MIRIAM MAUS: Totally echo that. There is this question about where is our lowest threshold? Is it really live one, a really interesting one? Because I mean, we're a society of publishers. Our mission is to expand the impact of physics and as a publisher, to give as many physicists the opportunity to share their research as we can. So that really speaks for the fact that if your content is publishable, we will publish it.
MIRIAM MAUS: But we haven't adjusted the process. You're absolutely right. It's the same process that you go through with a slightly different reviewer form. So then there is that debate of where does just OK stop and good start? And that threshold is it's really hard to institutionalize. I think individual editors will know it. They will know it when they look at it.
MIRIAM MAUS: But to do it at the scale that most sound science journals operate is really hard. And the other observation I would make about sound science is that the demand to publish in those journals is hugely increasing. These are the fastest growing submission. Trends we're seeing across our portfolio if it's a sound science. So there are people out there who want to publish because they need a publication.
MIRIAM MAUS: And the question for us is then, well, how much support can they expect for that?
DUSTIN SMITH: It's easier than ever to take a frontier, say fable class or Astra class model, apply it to open data sets basically legible to AI agents, and produce a paper end to end. You can say whether it's good or not. But ultimately, there's incremental advances in human understanding from applying that compute to data, and there's a structured output on the other side. So I'm curious where, maybe thinking IEEE, because it's more computer science relevant.
DUSTIN SMITH: If you're doing that, is that just for the provenance of preprints or GitHub or publishing by Twitter? And where's the bottom of where you're interested in?
DAWN MELLEY: Funny story. At IEEE, we have volunteers who are experts in AI machine learning, and they are the ones who are the most negative about using it in our publishing processes. So that tells me something. But, it doesn't. So it doesn't matter how you ask the question, my answer is going to be the same. We are having a really hard time finding that bottom cut off.
DUSTIN SMITH: And in some ways, it seems like intuition of people you respect, and domain experts is the best proxy for now. Absolutely. I mean, we count on the editors in chief, the editors that work under them and the reviewers to know their communities. Communities is a big word at IEEE, right? To know their communities well enough to be able to determine that this is an article that the community would like to read.
MIRIAM MAUS: Shall we ask Jonathan what he makes of those considerations? Because we, I think as clear, we agonize over what quality is, and we really go deep. And it's a philosophical question for many of us. When you hear us discussing that, what do you think?
JONATHAN WOAHN: I honestly end up having to say a lot of questions that like I just find personally curious about your process and how you guys are thinking about this. Because one of the very popular websites is archive.org. And this is where any AI researcher is going and just like-- I mean, they're getting flooded. And there's no approval process. It's just you go, and you upload it, and it's there.
JONATHAN WOAHN: But one of the things that's happening there is it's like the cycles are just starting to move so fast of like when the information gets out to when other people are wanting to get access to it. And so the question, that one of the things that I'm thinking about, as I'm hearing you guys talking about this is like if submissions are going up and you continue to trying to figure out where that floor is, but there's also this pressure of just like timelines, where it's like you can't have-- I mean, maybe in some domains, you can have really long feedback cycles where you can have multiple rounds going back and forth.
JONATHAN WOAHN: But as other adjacent fields of science continue to accelerate, how do you respond to that? Because by the time, if it gets published in a journal, the audience that might be most relevant for might have already moved on. And so that's like one of the questions that's going through my mind, is like how do you keep pace there?
DAWN MELLEY: So I can say that in the IEEE corpus, the fields that are moving the fastest tend to be the computer science based ones. And in those fields, it's more desirable to publish in a conference than a journal. I think for that very reason. Conferences are quick. And so that's where the pressure is in those fields. In journals, for me, and I think after 26 years of working with the volunteers, quality will always be the trump of everything else.
DAWN MELLEY: So if we have to slow down to ensure quality, then that's what we'll do.
JONATHAN WOAHN: So there's-- my former life, I was a lean expert. I worked in manufacturing plants. And so I very much came to-- I deployed a lot of manufacturing plants, the Toyota production system. And one of the, and please don't be offended by this, but one of the fallacies that a lot of the plants that we would go to is they would say time equates to quality. And the idea with the Toyota production system and lean is like, it's not about time, but it's about preventing errors.
JONATHAN WOAHN: And there's this concept that's called poka-yoke, which is like this is what a lot of the Detroit auto manufacturers adopted in following Toyota's rise was this idea of poka-yoke, where it's like instead of fixing things beforehand or afterwards, it's preventing mistakes from even being possible to be made in the beginning. And so it's like if you're installing a handle onto a car, if before, if there were five different ways that could get installed, like the poka-yoke process is to say, well, there's only one way this can get installed and to do it correctly.
JONATHAN WOAHN: And so with lean and in manufacturing, the idea is like it's not about time, but it's about trying to figure out how to manufacture errors out of the process. And so I guess that's one of the questions that I have in thinking through the submission processes is, as you guys were talking about injecting time into the process and injecting friction, there's not necessarily a direct correlation between quality.
JONATHAN WOAHN: So I'm just curious how you think how you think about that.
MIRIAM MAUS: I think this process of being lean and making sure that you frontload your QA is definitely in peer review. I mean, we break down the process from submission to acceptance like by the minute almost. So we really know how much time we spend at each stage and what we want the outcome to be. And this is where really technology, and tomorrow, Dustin and I will talk about Alchemist Review, and tools like that and others are really coming to the fore because they help us to basically support humans in getting it right the first time.
MIRIAM MAUS: So you have a system that does some of the heavy lifting, and then you reduce the amount of time that a person needs to look at it. And you do a load of that before anything even goes out to reviewers. So that when we get to the reviewers, we tell them don't have to worry about, checking references, formatting all that. Just tell us what is the contribution this piece of content makes to the field of, insert name of name of journal discipline.
DAWN MELLEY: Now if only we were making widgets, it would be great. No mistakes ever. But the slowdown comes in, Miriam is saying, we have all these tools. And so we're focusing the human resources on the things that they really need to focus on. But the issue there is, again, scale. And slowing down comes as a function of trying to scale. Because the human resources are limited.
DAWN MELLEY: So although we're getting their tasks more focused and they don't have to worry about as much as kind of peripheral stuff as they used to, there are only so many of them. So I can't take 10 reviewers and say, you still have to get through this many articles a day times 2. So the slowdown is in that backlog that has to work its way through the system. And we can't speed that up if the human resources are limited.
DAWN MELLEY: So where do we find the experts to scale that process? And that's where you would maintain or improve on the speed.
DUSTIN SMITH: Prepare your questions. I'll ask a couple more. But just so you have them in the can. And then Steph will be around with microphones. So both of you are highly reputable society publishers. But not everybody has that sort of community and reputation. Imagine there were publishing processes where you were so hard up for reviewers. Your median reviewer is mediocre, and your bottom quartile, reviewer is actually quite poor.
DUSTIN SMITH: And ultimately, that probably leaves you open to some sort of slop fraction making its way through. It's a polished research artifact. I'm curious, Jonathan, as you think about basically sizing and pricing portfolios of content, what is the potentially nefarious downstream effect of slop content, where a human direct to consumer might be able to say, this is slop, just sort of like seeing Claude slop in an email or slides.
DUSTIN SMITH: But robots ultimately are sort of like very fast consumers of pieces and parts.
JONATHAN WOAHN: Yeah. Well, what I love about both of your responses to my question here is the quality bar continues to be maintained at a certain level that reputation needs to continue to be intact. And so if you're engineering the waste out of the process at the time, really what you're doing is you're putting in a tremendous amount of effort upfront into the work that you're actually generating.
JONATHAN WOAHN: And a lot of that is what your institutions have done over the past, 20, 30, 40, 50 years, is building a reputation, building trust. And because in a Google world, your domain has gravitas. And so Google learned over time which sources were great, was able to build the classifiers to say for these types of questions, these are the authoritative sources for this. In an agentic world right now, literally, it's every agent out there is working off their own search algorithm.
JONATHAN WOAHN: They're working off their own prioritization. They're off their own classifiers, off their own scoring. And so what they are going to learn over time based on the human feedback is the pre-work that you are putting into the creation and the quality of the content will play itself out over time, and the agents will start to learn more broadly which sources are good and reputable sources based off of their human interaction and the behavior of how the people interact with it.
JONATHAN WOAHN: And if they start to request certain sources or certain journals or certain authors. But if it is the slop, then it's not going to get requested. It's not going to get it's not going to get picked up. And so you're not going to start to earn that gravitas. But in some ways, similar to Dawn's comment from earlier, it's like these are new tools that we just need to figure out how to make sure that that information gets communicated properly to the agents, so they can start making those determinations.
MIRIAM MAUS: And I think we have some agency here as publishers as well. We can be proactive. And there's a lot of discussion in the industry at the moment about trust markers and embedding trust markers. I'm a big fan of those initiatives, I have to say, because they win on several levels. They win on knowing whether you're a human or a robot or an AI agent, knowing the different categories of content.
MIRIAM MAUS: But they also win at the content production level because it gives authors, especially authors from less developed research community, a signal about, OK, this is what I can expect from this type of journal. And this is the kind of article that has gone through a certain process. So I think there is a win here for us working with technology and technology partners.
DUSTIN SMITH: Splendid. Do we have to give up some of the mix?
DAWN MELLEY: This is a lot for an introvert. Take my mic.
DUSTIN SMITH: Doing great, Dawn. Who we got? Hands up. Anybody? Bravery?
AUDIENCE: I found myself, as you all were discussing and thinking about the topic of quality, tracing back to COVID as one of the last kind of great pressure tests to quickly working through research and getting it out and making it accessible. So I think scholarly publishing learned very quickly how to operate at that speed and volume. And AI feels like another test of that, but something substantially more permanent that we're going to be working through.
AUDIENCE: So I'm just curious. I know you're not necessarily medical publishers. But what did we learn, maybe from the COVID era that's relevant to what we're working through and learning now with the adoption of AI and the increase in submissions, et cetera?
MIRIAM MAUS: Like I say, I'm not a medical publisher, so there may be other people in the room who have a better answer. But I think what we learned from COVID and the acceleration of research then is that if you get it right, the winds are great. If you get it wrong, the pitfalls are really serious. And I think this is what we should be taking into the AI era. It's so many opportunities, but the challenges are also real, and we need to be mindful of those.
MIRIAM MAUS: That doesn't mean we have to not pursue the opportunities. But there is something about responsibility here as well and understanding what is possible.
DAWN MELLEY: I also think that there's a resource issue just in successfully navigating through the change that AI is bringing in a technology sense. Because right now, I would say some of the AI tools we're using is making, they're making us less productive, right? We need to help them develop to the point where they make us more productive. And we also need to bring in different types of technology resources to help us navigate this era in the COVID sense.
DAWN MELLEY: IEEE kind of bucked the trend. We didn't see a huge increase in submissions through COVID. It happened afterwards.
DUSTIN SMITH: Yeah. I think COVID was 20% and then ChatGPT was 45. And then now, to the moon and beyond.
AUDIENCE: I have more of a comment. My name is Jasper Simons. I'm with the American Psychological Association. To gentlemen from the platforms, I find the analogy with manufacturing a terrible analogy in our industry. We're not trying to achieve a one standard and try to replicate it. So I really would love to hear more ideas about how you can support the novelty in science. Thank you.
DUSTIN SMITH: Support the novelty in science. I mean, I think quality in a manufacturing standard, much easier. You have stress and strain curves. You ultimately fatigue testing in industrial chemistry. You're taking retain samples. You're looking at adulteration rates. It's really collapsing to fewer dimensions. And I think everybody here would admit, like this is a messier enterprise of science.
DUSTIN SMITH: This is a collective of humans who are trying to validate our wisdom as a species. This is a big project that we're doing together. You can decompose, and we do, into things like rigor, novelty, and impact, and those are things which are increasingly machine capable activities that can assist humans basically do things like a first read and follow a set of procedures as if it were a human colleague.
DUSTIN SMITH: So that's a totally fundamentally different process than manufacturing. And it's kind of jokey that we're looking at parts per million here. Because that clearly fails. That's too simple of a standard. However, on the other side, to take a look and say these are my actual defect rates. Like this is our rate of fraud.
DUSTIN SMITH: Or ultimately, to what extent have you taken retained samples? There's a lot of trust in the community. How often do you take a retained sample of manuscripts or articles and do an end to end deep dive, where you have multiple experts spending a day apiece taking it apart into pieces to understand the quality of your processes? So I think there are things to learn. You obviously can't lift and shift directly from the manufacturing community.
DUSTIN SMITH:
JONATHAN WOAHN: To some degree, I agree on it is a terrible analogy. And one of the clients that I worked with in my previous life was an airline manufacturer or airplane manufacturer. They created airplanes. And literally every single airplane that came off that line is its own thing. And one of the things that we were able to do in working with them was creating checklists, being able to have a checklist of here are the gates, the dates and gates.
JONATHAN WOAHN: These are the things that we need to be able to pass through. And the analogy, as I think about this particular domain, and as platform providers, as part of what we can help provide is structure. And the reason that the structure becomes very relevant here is because without some kind of structure, everything becomes bespoke, and it becomes very-- it just becomes very slow to be able to get product out the door, to be able to get interoperability.
JONATHAN WOAHN: And so one of the ways, as a simple example in the age of AI, how we can think about this is think about what model context protocol or MCP has done for interaction between content and an agent. And so maybe we're not defining what the manufacturing process actually looks like, but being able to take a step back and say, what is the structure? What is the form? What is the way that communication can take place to help get the information out there, and to make sure that those trust markers get included and they get surface to the agents.
JONATHAN WOAHN: And so as I think about agentic access to scholarly content, that I've been operating against is there's kind of like five different layers to it. And it's like we need to be able to say, what are the trust markers of what is quality content? And you need to be able to tie those trust markers to the actual content itself. Those trust markers have to get delivered to the agents in order to be able to carry through who the author is or the provenance of that content.
JONATHAN WOAHN: The agents themselves have to be able to know how to interpret that content and how to interpret those trust markers. And finally, they need to know how to be able to give that reporting back. Like not just back to the platform, but back to the publisher, back to the agent, back to the authors to be able to complete that full cycle. And so part of the value that we can provide here is helping to think through that high level structure and make sure that it's actually enforced and actually deployed, so that the content that is being generated actually gets well represented in the way that you want it to within the environment where it's being consumed.
DUSTIN SMITH: Jessica Miles.
AUDIENCE: Jessica Miles, the informed frontier. I wanted to raise the point that quality and impact, I think as many of us in this room know, are not always immediately obvious during an article's publication or evaluation. Someone raised the Specter of COVID, and I myself was an editor of a microbiology journal during the COVID pandemic. And it was supremely challenging to understand what research was important or impactful while we're living through these novel conditions.
AUDIENCE: And so I wanted to ask if maybe this permissiveness that we've implicitly treated as a source of potential defects or problems with the research record is actually more of a feature than a bug. In some ways, we can guess what will move the field forward, or we might know more in some cases rather than others. But by letting maybe more in than we foresee being impactful, we allow the possibility that research, whose value might not have been recognized at the time of publication is ultimately added to the scientific record and can be used later.
AUDIENCE: So I would just throw that out there.
MIRIAM MAUS: Yeah. I mean, I entirely agree, and I think there are many, many examples in science and research where actually, something that didn't seem like much to start with then became a big thing. And that is definitely, I mean, as publishers and particularly as domain specific publishers, we have to create the space for that. And in fact, the members of our societies-- in physics, there's a big concern about blue sky research being sort of sidelined and everything has to be applied.
MIRIAM MAUS: That's a really important aspect of publishing that you make space for considerations and big, big ideas. So yeah. I entirely agree.
DAWN MELLEY: One of the things that's being talked about within IEEE is there's one camp. There's an editor in chief who's doing a research project right now to develop an AI reviewer. So the AI will actually do the review. And then others who say, but then what happens to novelty? So yeah. It's absolutely a challenge. And novelty always has been a challenge.
MIRIAM MAUS: I'm going to say something because I think it fits here, but I'm worried that I leave the stage and haven't said it. I think we really had this conversation very much on the publisher level. So we talked about publishers, but there is also a journal level. And I think we mustn't forget that there are different journals have different characteristics, different approaches, different criteria, different types of editors.
MIRIAM MAUS: And that's actually where kind of maybe this sort of the opportunity to create those spaces for discourse, that's where that sits in my mind.
DUSTIN SMITH: OK. One more, and then we're wrapping it up. I don't speak sign language, but I do speak Steph.
AUDIENCE: And it's not so much a comment or it's not so much a question as a comment. I'm Heather Kotula with Access Innovations. We're very proud to be a Silverchair Universe member. And we've addressed the scale issue for both peer review and conference proceedings for Plus, SPIE, and ASCO. And if you'd like to hear about it, I'd be glad to talk at the break.
DUSTIN SMITH: Straight up pitch. That's a strong move. All right. Yeah. Well, thank you, everybody. Is it 5 after the hour? OK. So this is break time. Do crosspollinate.
DUSTIN SMITH: Meet some folks. Get some refreshments. Be back seated by 5 after. Thanks so much. [APPLAUSE]
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode