Cadmore media player playing video If curation is the future, what actually works?
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
If curation is the future, what actually works?
Description:
If curation is the future, what actually works?
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/e8a6e42d-5e86-4a71-a1a2-dc58e8fab8f8/videoscrubberimages/Scrubber_10.jpg
Duration:
T00H54M29S
Embed URL:
https://stream.cadmore.media/player/e8a6e42d-5e86-4a71-a1a2-dc58e8fab8f8
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/e8a6e42d-5e86-4a71-a1a2-dc58e8fab8f8/Curation_1920x1080.mp4?sv=2019-02-02&sr=c&sig=8rG6UkGz25lnzSbpGD%2F%2FzWXkapAOjv9jXXvQpxdBNQg%3D&st=2024-12-21T13%3A16%3A33Z&se=2024-12-21T15%3A21%3A33Z&sp=r
Upload Date:
2020-11-18T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
WILL SCHWEITZER: The Platform Strategies Webinar Series, like our in-person meeting, explores the boundaries of information in publishing business and technology, and the strategies we employ to fulfill our missions and business objectives. Today, I'm joined by Claire Moulton, publisher of The Company of Biologists, Tasha Mellins-Cohen, director of technology and innovation at the Microbiology Society, and Bert Carelli, director of partnerships at TrendMD. And we're here today to discuss curation, which is a pretty broad topic, and can mean a lot of things.
WILL SCHWEITZER: But the Wellcome Trust, through a grant award that some of our panelists are involved in, provided a helpful definition that we'll use to ground our discussion today. That definition is, signaling the significance of published research outputs in an open and a transparent manner. Each of our panelists will present three or so slides on the topics.
WILL SCHWEITZER: And in the remarks, we'll touch on kind of their organization's approach to curation, and, perhaps, what they think works or doesn't. We'll reserve the bulk of our time for discussion and Q&A. You can use the chat function in Zoom to submit your questions at any point, which we'll hold until all the end of the panelists' slides. We'll first hear from Claire, then Tasha, then Burt. Claire, would you like to take it from here?
WILL SCHWEITZER:
CLAIRE MOULTON: I will just share my screen and start up. So, as Will said, I the publisher at The Company of Biologists. And we are a new publisher with Silverchair for 2020 on. So thank you for inviting me to the webinar. So I do think that community publishers play a significant role in curating content for their communities. Most commonly, that will be around special issues and subject collections in the content relevant to the journals.
CLAIRE MOULTON: But what I'm going to talk about today is a service that we launched called preLights, which is about selecting and writing about preprints. And this is prefixed with the whole of the biological sciences. They're not preprints that have been sent to our journals for publication. It would cover any preprint. So preLights is built around a community of early career researchers, and we call them preLighters.
CLAIRE MOULTON: And they've been very enthusiastic about preprints and transparency in communication. So a really great group to work with. And what they do is they select interesting or important preprints in their area of expertise. As I said, they can select that from the entire preprint literature. But we do notice that they predominantly select preprints from bioRxiv.
CLAIRE MOULTON: They then digest the preprint. They will explain key findings, the scientific background, and, importantly, they comment on why they personally found it interesting or why they selected it. So it's very much a manual curation. And it's generally a positive selection, with some constructive questioning around the content. So things that I think work well is, I think, that each post has a short tweetable summary.
CLAIRE MOULTON: And Twitter is the place where I would love to talk about preprints, so it provides a good source of browsable content for readers to see on Twitter before they come through to our site. Another thing that's worked well is that we've used the same subject categories as bioRxiv. And that's helped us create technology links between archives and preLights, which has been very useful. So for example, if you're on bioRxiv, and you're looking at a preprint that we have covered, you will see a link to preLights and the discussion of the preprint.
CLAIRE MOULTON: And then I also think the service provides a bit of value adds. So when the preLighter has written their post, they send it to the original preprint author with some questions about the research, whether they can tell them a bit more, or challenging some of their approaches. And the majority of the preprint authors do respond and provide comments back to our site.
CLAIRE MOULTON: So I think that that's quite interesting. Things that don't work so well, although the preprint authors do comment on the site, we don't get very much general commenting from readers. This doesn't surprise us. It's what we see in our other community sites, as well. But I think that that's something that we all hoped would develop more in the future. And I think that the other thing that maybe doesn't work quite so well is that each preLighter obviously selects preprints in their area of expertise.
CLAIRE MOULTON: And so, although we have quite a lot of preLighters, we have a little bit of patchy coverage. So I think that ways of navigating the content, you can obviously search the site. We add keywords for posts. You can use those subject categories, which, as I said, were the same as bioRxiv. And you could use those to select which preprint highlights you see on the site.
CLAIRE MOULTON: And you could also use them to select which preprint highlights you see in your email content alerts. We think that some readers also follow particular preLighters based on their subject speciality, which is quite nice to see. But, as I said, we did get a little bit of patchy coverage. So one of the things that we introduced was something that we call preLists.
CLAIRE MOULTON: So they were actually suggested by the preLighters. And they wanted to be able to highlight a larger number of preprints, and just not write quite as much about them, so that they could just cover more preprints with a lower amount of work. So these have actually been quite popular. We have a set of preLists that were on particular topics, in this case, antimicrobials. But we also have preLIsts that focus on the preprints that are featured in particular scientific meetings, and those are generally one-off lists.
CLAIRE MOULTON: But they have been very popular, as well. So, as it was mentioned, this year has been all about COVID, so I did want to mention how that has affected the site this year. And in fact, it's been very positive for us. So we had a huge number of preLighters join the team because people had more spare time during lockdown. So we now have about 250 active preLighters writing for us. And they also wrote more for us this year.
CLAIRE MOULTON: So we are just approaching our 1,000th post. And we had more of the preprint authors engaging, as well. And I guess I should also highlight that we've now got about 5,500 followers on Twitter. So they're getting a good amount of attention. You can imagine that we thought it would be a good idea to launch a preList about COVID preprints. And that was incredibly popular initially. But you will not be surprised to hear that we immediately lost control of the number of preprints being posted.
CLAIRE MOULTON: So a few of the preLighters actually moved to produce something separate from our site. And that is a site called covidpreprints.com. And we thought this was quite interesting because it presents the preprint alongside a timeline of the pandemic. And crucially, it calls out preprints that have been debunked, either because they overstated their claims, were too preliminary, or were potentially fraudulent.
CLAIRE MOULTON: So that's actually been a really interesting separate initiative. And I thought just to finish, I would highlight a new service coming out of eLife that they have just called Sciety. And this will be a gathering together of feeds from a number of services that either highlight or review preprints. So I think that that's one to keep an eye on.
CLAIRE MOULTON: So there we are. That's me finished. Thank you very much.
WILL SCHWEITZER: Thank you, Claire. Tasha?
TASHA MELLINS-COHEN: Apologies. Let me just unmute myself. So Will referenced that earlier this year, the Wellcome Trust and Howard Hughes Medical Institute launched the Learned Society Curation Awards. And it was very exciting for the Microbiology Society that we were successful in our application to convert one of the Society's journals, which is Access Microbiology, into an open research platform. And this along the lines of Faculty of 1,000 or Wellcome Open Research.
TASHA MELLINS-COHEN: From a values perspective, this fits very well with the Society. It offers us an opportunity to bring transparency to a traditionally very opaque process of peer review, and embrace open scholarship, which is very much the direction that we see publishing traveling in. It also helps us speed up the process of disseminating research, which maximizes the speed of communication as well as the potential impact and influence of each article.
TASHA MELLINS-COHEN: We've just heard from Claire about preLights, and how valuable that is proving to be as a way to communicate research at an early stage. The reason that we are thinking of this as anti-curation, as it were, is that a traditional journal was hyper-selective. They had to confine the published research to the bound contents of a print journal. Modern journals have somewhat broken those boundaries.
TASHA MELLINS-COHEN: And we now have many journals, many articles which incorporate open data, open methodology, so on and so forth. But we see an open research platform as a new form of product. It's not just a journal. It gives us an opportunity, for example, to incorporate posters from our conferences, to potentially publish the theses of our members if they wish to do so.
TASHA MELLINS-COHEN: There's so many more things that can be incorporated into an open research platform that it becomes less about curation and more about layering and anti-curation. So as you can see here, this is a very simplified, but very traditional workflow for a journal, well, traditional in the sense of preprints having come into play about 20 years ago.
TASHA MELLINS-COHEN: Everyone could see a preprint. We know that, we appreciate that. But from the point of preprint to the point of formal publication is very opaque. A technical check will happen, but if an editor decides at that point to reject the article, nobody sees that information. Nobody sees why that rejection has happened. Similarly, peer review, nobody really sees the peer review reports except for the editors and the authors themselves.
TASHA MELLINS-COHEN: You don't have that level of transparency. And only after many rounds of review and revision and editing, and is a very highly curated process, will the real world, the world outside of the editors and the reviewers and the authors, see the final version of the article. So most research, most of the process of research is curated, and very carefully so. For an open research platform, the workflow changes.
TASHA MELLINS-COHEN: And it looks a little bit more like this. So there are many, many points where versions and reviews of those versions are posted. So our intention is that the point an article is submitted-- It might well have already gone onto bioRxiv. That's entirely up to the authors-- we will run machine learning technology peer review. So I suppose not truly a peer because it's not human. But we will be looking at things like SciScore and authenticate to make sure that the research meets certain basic standards.
TASHA MELLINS-COHEN: The initial preprint will then be posted to our platform. And that's our standard microbiologyresearch.org platform with the AI review reports made available alongside it. Transparent peer review will be managed through our editorial manager system. We're not making use of any kind of bespoke software here. It's all the stuff that we already have in our technology stack.
TASHA MELLINS-COHEN: And again, the reviews will be posted once they've been received. And the revised article will be submitted. And we'll see a new version of the preprint, and so on and so forth. Everything will include relevant persistent identifiers, whether that's DOI for all of the versions of the article and for all of the peer review reports, links to open data, use of funding data, usage of Research Organization Registry information, use of ORCID, all of that will be incorporated at every point in the process.
TASHA MELLINS-COHEN: So in this way, we get a very clear very transparent, very versioned view of the research process. Now, this bit is anti-curation. What comes next is a bit different. After this, with this formal publication of the the version that the reviewers and the editors and the authors agree is acceptable, the communities around the Microbiology Society and, indeed, beyond the microbiology society, will have an opportunity to create, effectively, overlay journals or reinvented curation, but reinvented curation over the top of a complete transparent process.
TASHA MELLINS-COHEN: I thin k that is it. Yes. I will hand over to Bert.
WILL SCHWEITZER: Thank you very much.
BERT CARELLI: Excellent. Thanks, Tasha. Thanks, Will. Let me just. So TrendMD is a technology company. So I'm going to focus a bit on technology. And to provide a bit of oversimplified context here, we can look at technology as having played an evolving and disruptive role in the curation of scholarly content.
BERT CARELLI: Up until the 1960s, curation was the province of editors in libraries. Manuscripts were selected in issues assembled by editors, in libraries acquired and organized by journal title, volume, and issue, discovering new advances in one's own field, and first acquiring access to the key journals, and then systematically browsing tables of contents, or in a few fields, like medicine, browsing scholarly indexes that could be months or years out of date.
BERT CARELLI: These methods for keeping up with new discoveries were highly inefficient, and often prohibitively expensive. NASA learned during the early days of the space race that it was often more expensive to figure out if an experiment had already been done than it was to simply do it again, from scratch. In fact, that realization led to the creation and funding of one of the first information retrieval databases at Lockheed Missiles and Space, and the era of digital creation curation was launched.
BERT CARELLI: Database era ushered in major changes in the way scholarly information could be curated. Instead of pouring through journal by journal, issue by issue, a researcher, usually with the help of a librarian skilled in the specialized language of command line search, could look at the entire archive of a journal, or search a citation database using subject taxonomies like MeSH, and curate lists of appropriate articles for the researcher to review.
BERT CARELLI: Flashing forward to the first two decades of the 21st century, web technology has given us the ubiquitous search box, from which virtually all scholarly material can be accessed, and where curation is effectively done at the article level. A keyword search on any subject instantly "curates"-- I'm using air quotes here-- a list.
BERT CARELLI: The quality of that list can vary widely. For example, you might see a peer reviewed article from an established journal on a subject like the flu vaccine sitting next to a conspiracy monger's post claiming vaccines can cause autism. But accepting that the serious researcher can deal with its obvious flaws, we can see that curation, the Google search does fulfill another purpose. Sometimes you're not just looking for a specific piece of content, which Google's obviously very good at providing.
BERT CARELLI: Sometimes a researcher also wants to see what other related content might be interesting or useful. In this case, the search results page becomes not the end, but a key step in the larger process of discovery. In their survey, "How Readers Discover Content in Scholarly Publications," Tracy Gardner and Simon Inger found that users are also likely to browse other articles that surface in a search.
BERT CARELLI: So the need to browse to learn what you did not think to ask, maybe allowing for a little serendipity, is still a vital part of the research process. And this is where this related articles function comes into play, which is TrendMD's business. Going back to the study that I just mentioned, which has been done every three years since 2005, users were asked what features of published websites they use most often.
BERT CARELLI: Over the five iterations of that survey, the related articles feature showed consistently increasing popularity, while other website features, like publisher produced news, site search, saved search, alerting, declined. TrendMD's recommendations widget uses behavioral data and AI technology to create another type of browsing experience for the reader, suggesting additional articles based on what they are reading at the moment, what other users like them have read, and what they've read in the past.
BERT CARELLI: Unlike a search-generated browse list, which contains thousands of items closely related to the search terms, TrendMD's focus is to curate a short list of related items with the highest likelihood of being of interest to the reader. It's similar to the way that Amazon works. If you buy a teapot through Amazon, the site's not going to show you more teapots.
BERT CARELLI: You'll see products that other buyers of teapots have bought, like recipe books, teacups, or tea. In TrendMD's case, a medical practitioner reading an article in The BMJ on smoking hazards, may see recommended articles on lung cancer. While a public health policy specialist may see articles on secondhand smoke, and a biochemical researcher may see articles on tumor formation. The other aspect of TrendMD's approach is that these recommendations are selected from the largest possible collection of content.
BERT CARELLI: Half of the recommendations are from the same journal or publisher as the article being read. And the other half are from a network of over 5,000 scholarly sites. Result is that the user sees content from journals and publishers that they might otherwise not be familiar with. And it enables authors and publishers to reach audiences that will truly value their content. Number of case studies, including a 12-month randomized control trial that demonstrated that articles promoted by TrendMD were 50% more likely to be cited, have shown the value that this feature provides for publishers, authors, and researchers.
BERT CARELLI: Back to Will.
WILL SCHWEITZER: Thank you very much. So just want to remind the audience, you all can submit questions via the Q&A, which you'll find at the bottom of your Zoom screen. And we have just a few questions to kick off our discussion. And Claire, I thought I'd come back to you first. And you've already answered two audience questions around who are the preLighters, and how were they selected and trained. But do you know if the preLighters are using any technology to either identify, or select, or even summarize preprints they may find?
WILL SCHWEITZER: Is that something you all are looking into?
CLAIRE MOULTON: No, it's not actually. So I think it's a really good question. But is a very manual human process at the moment. And I think that we did have in mind that we might try those sorts of things in the future. But I think people quite like the real personal touch. And one of the things that has really enthused people about this product is actually the early career community. And they are really keen on preprints.
CLAIRE MOULTON: They write preprints with meta-analysis about preprints. And they do lots of presentations about preprints. And so it's very much a human endeavor at the moment.
WILL SCHWEITZER: Thank you. So Tasha, I'm going to purposefully ask a pointed question here to maybe generate some more discussion? And, actually, I have two questions. And one is, do you feel this ant-curation approach exacerbates the glut of information that's out there?
TASHA MELLINS-COHEN: I think the flood of information is happening anyway. The anti-curation approach, by introducing transparency and providing levels of reporting even on that very first preprint, which simply doesn't exist in bioRxiv, helps readers to understand that what they're seeing might not be the final version, but has been through some level of analysis, some level of checking.
TASHA MELLINS-COHEN: Interestingly, when Claire mentioned early career researchers, Access Microbiology as a journal was originally created specifically in response to Early Career Microbiology Forum saying that they needed a place to publish small papers, replication studies, negative results, and so forth. And actually, not only was that journal's scope defined by the early career group, but the editorial board is made up primarily of early career mentees, paired with very experienced mentors.
TASHA MELLINS-COHEN: And that group has proven to be very receptive to different ways of working, including this kind of anti-curation approach. So it was it was them. It was the early career group who were really instrumental in us applying for the Wellcome Grant. And they're the ones who are saying, this will make our research easier, better because we will be able to see what is valid immediately.
WILL SCHWEITZER: Oh that's great. And then I know it's kind of relatively early days for the journal over the open resource platform. But are you seeing any user data that suggest readers are accessing and using either the peer review commentary, the decision letters, or the material around the final version record article?
TASHA MELLINS-COHEN: So we aren't live yet. We're still very much in build. It's very early days. But part of the reason that we went ahead with this is that, actually, if you look F1,000 or Wellcome Open Research, you can see how much traction the reviewer reports are getting on those platforms. So you can see how many times those have been read. Now with any download statistic, any usage statistic, you don't know for sure if somebody's actually read the piece, or if they've just opened it.
TASHA MELLINS-COHEN: But we know that people are making use of that. They are visiting the peer review pages, which gave us confidence that if we were to do this, we would be able to add value to the publication process by adding transparency.
WILL SCHWEITZER: That's excellent. I think it's been, I would say, an increasing trend we've within this Silverchair community that publishers are bringing new questions to their platform analytics and their usage data.
TASHA MELLINS-COHEN: Yeah.
WILL SCHWEITZER: It seems like five years ago, we were all taking for granted that folks were coming in from Google or a large-scale search engine, finding the PDF that they wanted, and leaving. And I think as we dig in further, we're finding there's really valuable use cases and services we can add on.
TASHA MELLINS-COHEN: Yeah.
WILL SCHWEITZER: Bert, my next question is for you, which is you provided some great data around how frequently the TrendMD widget and recommendations are used on platform, some potential citation effects for the widget, which is really great data. But I'm thinking about editorial stakeholders in the process, and the efficacy of the recommendations, and if you have any data, anecdotal or otherwise, that's compares, say, TrendMD's recommendations versus editorial or human-curated links in the process.
BERT CARELLI: Well, yeah. Thanks, Will, for that question. At your former organization, we went through a very lengthy trial, starting out on some of the newer journals, the smaller journals. And the editors weighed in heavily in the early days about the TrendMD recommendations. And the conclusion, and why we're now on all of the AAAS journals, was that they actually preferred the engagement that was being shown for the TrendMD recommendations.
BERT CARELLI: And it's actually not surprising because from my experience back working with publishers over the years, I've noticed that publishers are not so attentive to the use of features on their website. And one of the things that has been on websites for many years is some type of related articles functionality. And often, it was one of the least used functions even when editorial staff spent a lot of hours trying to curate that list of, here's additional content that the user might be interested in.
BERT CARELLI: So being able to come up with it, being able to offer an automated solution was helpful, certainly from a cost perspective for the print publishers. But also, the engagement that was provided clearly indicated that people were using them. And that's really what TrendMD functions on is clickthrough rate as the key indicator of quality. If people are engaging with the widget, if they're clicking on the articles, then it's a pretty good indication that there must be some quality there.
BERT CARELLI: So it's really leaving it up to the users, the wisdom of crowds, if you want to think of it that way, to dictate whether the quality is there. And so far, it's been consistently high.
WILL SCHWEITZER: Thank you. So I have one last question before you all if you have questions for each other. And then we can turn to some audience questions, which is we've brought up kind of COVID. And we've seen a proliferation of activity around preprints and COVID. Chuckling at this point may be a good coping or safety mechanism. But earlier today, Kent Anderson in The Geyser wrote about the potential problems of preprint servers, and essentially disinformation.
WILL SCHWEITZER: And I think it points this really interesting tension of the role of curation, and essentially signaling significant and valid research. And if I were to put an editorial comment on preLights, or define communities that I see, it seems to be working with a closed community. So it may be working within a group of several thousand life sciences researchers, but we don't seem to have effective curation mechanisms, yet, at scale, or at the seam of science and the broader public, or, perhaps, even science in the policymaking community.
WILL SCHWEITZER: I was wondering if you all have any thoughts about this. Are you seeing anything in your communities with your programs that might give possible solutions or indications for solutions for this problem about curating research or even early stage research for the problems facing us today?
TASHA MELLINS-COHEN: May I?
WILL SCHWEITZER: Please.
TASHA MELLINS-COHEN: I would argue that there are also problems with research that is published in a journal post peer review. One of the biggest public health crises we faced-- this would be prior to COVID-- is the damage done by the Wakefield paper, which was published after peer review in a very well-respected journal. And it has created a hideous situation, where many people are rejecting vaccination, which is a perfectly safe and sensible public health measure.
TASHA MELLINS-COHEN: And it's allowing terrible diseases to spread again in communities that should not have to face it. So, yes, I absolutely agree that the proliferation of non-peer-reviewed research being available to everybody could create a problem with disinformation I don't think the peer review is necessarily the only way to solve it. But I do think that the application of the sorts of machine learning tools that we are proposing to use for preprints could help to flag, for example, there is a methodology problem here, or these authors have not declared their conflicts of interest, and give people some context and some guidance as to where there may be flaws in that manuscript.
TASHA MELLINS-COHEN: But we have for decades said, peer review means something is valid. And that is not necessarily the case. And Kent and I have had a vocal argument about that, so [LAUGHS] I won't carry on.
CLAIRE MOULTON: I would agree with that. I think that posting or publishing, wherever if you are in the cycle, it's still just part of the beginning of the discussion of the science. People want to discuss it, they want to reproduce it, and they to challenge it. I think that one of the interesting things that is going to come out of this past year is that I think that the people who were in favor of preprints will be feel even more strongly about it because they can see that the rapid fabrication of work has played a very important role this year around COVID.
CLAIRE MOULTON: And the people who have had concerns about preprints will feel more strongly about that because they will have seen preprints that could potentially cause harm or, certainly, diverted resources. So I suspect that the viewpoints may have become even more opposite.
WILL SCHWEITZER: Seemingly with all things these days. Bert, any thoughts from you?
BERT CARELLI: Well, I think that there's a certain philosophical perspective here. When Cold Spring Harbor first launched bioRxiv, I was working with them to help them launch the preprint site. And, to be honest, I had some skepticism after working with scholarly journals, with the peer review journals for so many years, and knowing how editors were so protective of their space, of their choices, of what to publish.
BERT CARELLI: But I became a convert, to be honest, especially seeing the growth and the enthusiasm. I have a son, who at the time was in graduate school and working in biochemistry, and I often used his group, his peers to show me early examples, even before bioRxiv launched. This is the type of thing-- This is what is being attempted here. And the overall enthusiasm of those young people, of those early career scientists, who are now know actually in there the middle of their careers, was overwhelming.
BERT CARELLI: And I think that there's a sense that if we can trust the process-- There are going to be these outliers, there are going to be the Wakefield papers, and there are going to be some of the COVID papers that have been really scandalous. But there's a certain wisdom in crowds that we're almost committed to that today anyway because we can't deny that the tides are rising around us all the time of information.
BERT CARELLI: So we have to trust that there's an audience there that has the ability to make some critical choices about things. So, I don't know, that that's my sense in why I feel like a technology solution, you can't turn the clock on it. Once the genie is out of the bottle-- to mix many metaphors here.
WILL SCHWEITZER: Still a trust or hope for audience discernment. So before turning to some audience questions, we have three in so far, and some of them are a bit philosophical, I wanted to see if you all had any questions for each other.
CLAIRE MOULTON: I do have a question for Tasha, actually. It's quite a practical one. But when I was talking about your new workflow with my team, they were interested in, what happens if a preprint comes to you, goes through the review, and then the author chooses not to do the revisions that you've requested? Can they then continue and take that paper elsewhere? How does it work?
TASHA MELLINS-COHEN: Well, effectively, it is a preprint. So if they really don't want to continue, then, yes, they can take that elsewhere. And they can say, this is my preprint, it's in the open research platform, I have chosen not to finish the publication process there. There's no reason that we couldn't let that happen. From the conversations that we've had so far, the researchers that we've spoken to have sort of said, oh, why would I do that?
TASHA MELLINS-COHEN: If I've gone through all that process, and I've gone through the reviews with you, why would I take my article elsewhere and start again? So I don't foresee it necessarily happening particularly frequently, unless and until there is a pretty broad acceptance of transfer of peer reviews, which isn't hugely popular at the moment. And there's a lot of editors that I work with personally who've over the years said, I wouldn't trust somebody else's peer reviewers.
TASHA MELLINS-COHEN: Why? If the peer reviewer's good, the peer reviewer's good. So, yeah. Hypothetically, people could get partway through the process and pull out at that point. There's also a possibility that they get partway through the process and refuse to meet the requirements of the editor to amend their paper to make it satisfactory.
TASHA MELLINS-COHEN: And at that point, it would remain simply a posted preprint. We wouldn't go through the production process, which I think is possibly more likely to happen.
WILL SCHWEITZER: Any other questions for each other?
BERT CARELLI: Yeah, actually I have a question really probably more for Claire, but also Tasha. I've been talking with a number of companies that work with automated summarization and sentiment analysis, and I'm curious whether you've discussed or considered adding this type of thing to somehow scale up the process? Especially, Claire, it's impressive the numbers that you mentioned of the preLights involvement.
BERT CARELLI: But still, it's a very small fraction of the preprints that are there out there.
CLAIRE MOULTON: As I say, it's a really good point. It is one that we have discussed. I think that we're less likely to do it on preLights because I think that one of the parts of the post is the, why did I personally choose this preprint? And what are the questions that I want to ask to the print author? And be able to follow up in more detail with those. I think that where we more interested is-- So often, with research articles, we write a research highlight, so just a short summary the basically encourages readers to go and read the article.
CLAIRE MOULTON: And that's the sort of work that our in-house team currently write. And I can see a time saving for using an automated process for that. And we would still have I think that certainly in the early days, would, certainly then have a person checking that before it went live. But, yeah, we are interested in the idea.
TASHA MELLINS-COHEN: So we are actually going to be using one of those tools as part of our initial pass, initial automated pass through each preprint, so that we can give people an idea of what's in there. Because why not?
WILL SCHWEITZER: Great. It'll be exciting to see how that gets on. This next question's from the audience. And it's a bit of a philosophical one, which is, is there a distinction between recommendation and curation? There's probably a related point here about how do we label those recommendations or curated links for end users.
WILL SCHWEITZER:
TASHA MELLINS-COHEN: I think I would draw a distinction because there are articles that I might collect together for a particular purpose, and I would consider that to be curation. But the purpose might not be positive. It might not be something I would recommend that somebody read. I might be collecting it together to debunk it or to another way challenge it. I don't know how the others feel about that.
CLAIRE MOULTON: Yeah, I would also distinguish between them. I think, yes, curation is more the collecting together than a recommendation. But I suspect the most curated collections will be recommendations. But I think they are different.
TASHA MELLINS-COHEN: Yeah, and a reference list is a curated list. But it's often, this is fake, this is negative. Sorry, Bert.
BERT CARELLI: Yeah, well TrendMD comes at this issue, and it's a big question for us, certainly, that we think about all the time from really a slightly different perspective. The problem that we set out to solve was really one more for publishers and authors who wanted to have their content exposed to broader audiences. So the purpose was somewhat different. And I really appreciate, Tasha, your mentioning curation for other kinds of uses.
BERT CARELLI: So we're really talking about really different intent more than the difference in what the feature actually just throws out there on a site. And I think, probably, there could be more clarity around that from the perspective of labeling these lists of recommendations as recommendations that are coming from a particular perspective, that these are what the author and publisher wants you to see, which is what you're going to expect to find on a journal site.
BERT CARELLI: These are the issues that have emerged out of the content, maybe, perhaps, but from a different perspective, from the more qualitative analysis of the site. That's kind of where I was coming from about the question about automated summarization and sentiment analysis. I wonder if there are multiple opportunities for recommendation technologies that could be applied in these different ways.
CLAIRE MOULTON: And I think recommendation doesn't always mean that you agree with the finding. It could be that you have back to back papers argue two sides of an argument. But you would recommend that people read both of them to see the two sides of the argument.
WILL SCHWEITZER: That's a really good point, great. So this is a related question, and it is, how do the goals of users and the goals of publishers come into play in curation? And do those goals conflict at times?
CLAIRE MOULTON: I think that when you are launching something like this, you're aiming it at the user. Of course you want to, perhaps, promote the fact that you have some strong content in an area, or something like that. Or you want to present some content because you want more submissions in that area. But you are always thinking about the user. That's why you produce it.
CLAIRE MOULTON: If people aren't reading it, then it hasn't succeeded. I'm not sure that I would say that aims conflict. I'm sure that publishers get it wrong sometimes, and produce a collection that the readers aren't as interested in as we'd hoped. For example, I think that when they announce the Nobel Prize winners, lots of publishers suddenly go, oh, yes, we've got some papers on that! And we've got a fantastic paper by that particular author.
CLAIRE MOULTON: And we pool collections together, and then not always very successful. We feel that we need to showcase the work. But we do at least always have the user in mind.
TASHA MELLINS-COHEN: I think, as well, the idea of "the user" is really interesting to me because we have many different communities we are trying to serve simultaneously. So if you're a publisher, you need to serve authors, reviewers, editors, and librarians, and funders. If you're a society, you also need to think about your members, and the different types of members. So the idea of a user, I think, generally, we're trying to serve a specific group of users, not necessarily everybody, all the time.
TASHA MELLINS-COHEN: So, yes, I'm sure there are times when a publisher is launching a new journal, and a library is just going, god, we don't need another journal. Please not another journal. That's just going to cost more money. But the editors, the people who want to launch the journal, are saying, there is a niche and nobody is publishing in this area.
TASHA MELLINS-COHEN: So, in that instance, we are curating content as a publisher with conflicting demands from the two different types of user.
BERT CARELLI: And users are not necessarily monolithic in their process. When you think about, well, again, coming from the perspective of TrendMD, where we find the most usefulness, where users really raise their hands and say, yeah, this was really useful to us, is when they are looking for that piece of serendipity. They're looking for that. One of the other topics that I may have missed, so it's, if you think of it as an early part of the research process, maybe it's the PhD student who is fishing around, like, where am I going to focus the next few years of my career?
BERT CARELLI: And it's also, to some degree, the person that runs a lab is looking, like, what are the experiments that I should be planning for in the next two years? So it's more keeping the wide end of the funnel as open as possible. Whereas, you know you could really imagine curation at a later stage, where you really need to be conscious of, is someone else publishing a paper that's close to this particular topic?
BERT CARELLI: Are these compounds being used in another way that I should be aware of before I publish? So I think it plays a role in all those different stages. So it's very much a moving target, is what the user really is interested in.
CLAIRE MOULTON: I think you're spot on there, actually. I think that it's finding the balance between wide and narrow. And, certainly, the feedback that we get from researchers is that whether a list is manually curated or automated, they worry that it's not going to include the one thing that they would have filed if they put the list together. But at the same time, if you always do a broad list, then they're not going to find all of the pieces quite so interesting.
CLAIRE MOULTON: And it is a really crucial part of the puzzle.
WILL SCHWEITZER: This question is from our friend Richard Wynne, which is, does curation include establishing the specific contributions of each author thinking of credit attribution?
TASHA MELLINS-COHEN: It could. It doesn't always. In practice, in reality, I think it would be a wonderful thing to be able to say, yes, every author on this paper has a credit tag. But it doesn't always happen.
WILL SCHWEITZER: And I guess that brings up a slightly broader point, which is, how should we be valuing or recognizing the act of curation? Is that a similar level of importance, say, as authorship? I think as the credit taxonomy is designed to recognize. Any thoughts on that?
BERT CARELLI: Well, it comes back to the value of libraries, I think, at its core. The library was always responsible for curation. And the value of librarians themselves and their expertise to field. The advantage that a librarian has is of impartiality. Their only skin in the game is to serve the user, the researcher.
BERT CARELLI: So I think that that's where the value ultimately should be. That's the attention that we should always be putting on this whole process.
TASHA MELLINS-COHEN: There's also a question in there about recognizing review and editorial contributions because, at the moment, again, most of that is hidden, mostly it's not named. Even if you're providing ORCID feeds with information about who's contributed to peer review, It's not really clear. And that is a part of the curation process. That editorial selection and peer review is a part of the curation process.
TASHA MELLINS-COHEN: And we don't really have a proper taxonomy around that. So if somebody has done, for example, a statistical review or a methods review versus a general peer review, we have no really clear way, within CRediT, or any other taxonomy, to define that. And we don't describe our editorial processes, particularly in a standardized way. So there is a question there about the role of taxonomy for the process of curation, as well as for credit for the authors.
TASHA MELLINS-COHEN:
WILL SCHWEITZER: And I think for our last audience question, Claire, this one is for you. And I'm trying to synthesize on the fly here. But it seems like a company biologist is making it an investment either in human time or dollars in the summaries. And then whether you have thought about repackaging those summaries for audience, and whether that would have revenue-generating potential.
CLAIRE MOULTON: So we did decide to invest in this area. And we don't currently have any commercial revenue-generating ideas at all. We've specifically set those aside, so that we could just develop the project, and take ideas from the preLighters themselves, and invest in those, as well. So we've done quite a few of those sorts of things. I think it depends on the nature of the business. We have tried to set ourselves as a community publisher.
CLAIRE MOULTON: We have a number of community sites. We have one called the Node, which is for developmental biologists, so it's associated with a particular journal. And we have one which we launched just this year, called focal plane, which is about imaging and microscopy. And that relates to all of our journals in some way. And we've really just invested in those. They don't cost a huge amount of money.
CLAIRE MOULTON: And we feel that they do provide a big benefit to the community. And so we decided it was part of our-- We're not a society, but we are a community publisher. And we just thought it was part of our community service to provide those forums.
WILL SCHWEITZER: And there's a second part to this question, which is, do you have any comparative data, say, comparing how these summaries are used or how preLight is used relative to, say, your original peer-reviewed research and in your journals?
CLAIRE MOULTON: Well, I don't think we've really tried to compare them as such. Clearly, they have a very big Twitter presence. All journals have a Twitter presence, as well. But this is especially hot on Twitter. And I think that they will be the full piece will be read fewer times than one of our published articles because I'm sure that a lot of people read that the tweetable summary, and they always feel like they read the article.
CLAIRE MOULTON: A little bit like when someone spots an article on a table of contents, they think they've read it. But I think that it does create an excitement and a visibility for the preLighters and what they've done. A lot of people tell me that they use preLights a lot. And so you know I think that the warm feeling that people have is one of the benefits of having produced this service.
CLAIRE MOULTON: And one of the reasons-- Obviously, there's always a little bit of a selfish reason for launching a product-- But we did feel that preprints were hot. We wanted to show that our journals were preprint friendly, we accept reprints, we do transparent peer review, we do a lot of things around transparency and openness. And we thought that this would, in some ways, reflect on us as an organization, that we'd embraced preprints and early career researchers.
CLAIRE MOULTON: And that has definitely paid off. So if you were looking for sort of a selfish nugget in what is really a community initiative, that would be it. I think it made us look cool.
WILL SCHWEITZER: Thank you. Well, do you have any last questions for each other?
TASHA MELLINS-COHEN: Not for me.
CLAIRE MOULTON: No, good discussion.
WILL SCHWEITZER: Well, first, a very big thanks to you, Claire, and Tasha, and Bert, for joining our panel today. And a really big thanks to our audience for joining us and participating. As my colleague, Stephanie, mentioned at the beginning of the webinar, registrants and attendees will receive an email with a link to this recording, and a link to a participant survey. Your feedback will be really helpful to us as we plan for 2021.
WILL SCHWEITZER: So please, take a moment to complete it. I'd really appreciate it. So thank you all again. And we'll see you on a webinar, or perhaps, even at a live event sometime soon.
BERT CARELLI: [LAUGHS] Let's hope so.
TASHA MELLINS-COHEN: Thanks and goodbye.
CLAIRE MOULTON: Thank you, bye.
BERT CARELLI: Thank you both. Bye, Claire. Bye, Tasha.
Segment:0 .
WILL SCHWEITZER: The Platform Strategies Webinar Series, like our in-person meeting, explores the boundaries of information in publishing business and technology, and the strategies we employ to fulfill our missions and business objectives. Today, I'm joined by Claire Moulton, publisher of The Company of Biologists, Tasha Mellins-Cohen, director of technology and innovation at the Microbiology Society, and Bert Carelli, director of partnerships at TrendMD. And we're here today to discuss curation, which is a pretty broad topic, and can mean a lot of things.
WILL SCHWEITZER: But the Wellcome Trust, through a grant award that some of our panelists are involved in, provided a helpful definition that we'll use to ground our discussion today. That definition is, signaling the significance of published research outputs in an open and a transparent manner. Each of our panelists will present three or so slides on the topics.
WILL SCHWEITZER: And in the remarks, we'll touch on kind of their organization's approach to curation, and, perhaps, what they think works or doesn't. We'll reserve the bulk of our time for discussion and Q&A. You can use the chat function in Zoom to submit your questions at any point, which we'll hold until all the end of the panelists' slides. We'll first hear from Claire, then Tasha, then Burt. Claire, would you like to take it from here?
WILL SCHWEITZER:
CLAIRE MOULTON: I will just share my screen and start up. So, as Will said, I the publisher at The Company of Biologists. And we are a new publisher with Silverchair for 2020 on. So thank you for inviting me to the webinar. So I do think that community publishers play a significant role in curating content for their communities. Most commonly, that will be around special issues and subject collections in the content relevant to the journals.
CLAIRE MOULTON: But what I'm going to talk about today is a service that we launched called preLights, which is about selecting and writing about preprints. And this is prefixed with the whole of the biological sciences. They're not preprints that have been sent to our journals for publication. It would cover any preprint. So preLights is built around a community of early career researchers, and we call them preLighters.
CLAIRE MOULTON: And they've been very enthusiastic about preprints and transparency in communication. So a really great group to work with. And what they do is they select interesting or important preprints in their area of expertise. As I said, they can select that from the entire preprint literature. But we do notice that they predominantly select preprints from bioRxiv.
CLAIRE MOULTON: They then digest the preprint. They will explain key findings, the scientific background, and, importantly, they comment on why they personally found it interesting or why they selected it. So it's very much a manual curation. And it's generally a positive selection, with some constructive questioning around the content. So things that I think work well is, I think, that each post has a short tweetable summary.
CLAIRE MOULTON: And Twitter is the place where I would love to talk about preprints, so it provides a good source of browsable content for readers to see on Twitter before they come through to our site. Another thing that's worked well is that we've used the same subject categories as bioRxiv. And that's helped us create technology links between archives and preLights, which has been very useful. So for example, if you're on bioRxiv, and you're looking at a preprint that we have covered, you will see a link to preLights and the discussion of the preprint.
CLAIRE MOULTON: And then I also think the service provides a bit of value adds. So when the preLighter has written their post, they send it to the original preprint author with some questions about the research, whether they can tell them a bit more, or challenging some of their approaches. And the majority of the preprint authors do respond and provide comments back to our site.
CLAIRE MOULTON: So I think that that's quite interesting. Things that don't work so well, although the preprint authors do comment on the site, we don't get very much general commenting from readers. This doesn't surprise us. It's what we see in our other community sites, as well. But I think that that's something that we all hoped would develop more in the future. And I think that the other thing that maybe doesn't work quite so well is that each preLighter obviously selects preprints in their area of expertise.
CLAIRE MOULTON: And so, although we have quite a lot of preLighters, we have a little bit of patchy coverage. So I think that ways of navigating the content, you can obviously search the site. We add keywords for posts. You can use those subject categories, which, as I said, were the same as bioRxiv. And you could use those to select which preprint highlights you see on the site.
CLAIRE MOULTON: And you could also use them to select which preprint highlights you see in your email content alerts. We think that some readers also follow particular preLighters based on their subject speciality, which is quite nice to see. But, as I said, we did get a little bit of patchy coverage. So one of the things that we introduced was something that we call preLists.
CLAIRE MOULTON: So they were actually suggested by the preLighters. And they wanted to be able to highlight a larger number of preprints, and just not write quite as much about them, so that they could just cover more preprints with a lower amount of work. So these have actually been quite popular. We have a set of preLists that were on particular topics, in this case, antimicrobials. But we also have preLIsts that focus on the preprints that are featured in particular scientific meetings, and those are generally one-off lists.
CLAIRE MOULTON: But they have been very popular, as well. So, as it was mentioned, this year has been all about COVID, so I did want to mention how that has affected the site this year. And in fact, it's been very positive for us. So we had a huge number of preLighters join the team because people had more spare time during lockdown. So we now have about 250 active preLighters writing for us. And they also wrote more for us this year.
CLAIRE MOULTON: So we are just approaching our 1,000th post. And we had more of the preprint authors engaging, as well. And I guess I should also highlight that we've now got about 5,500 followers on Twitter. So they're getting a good amount of attention. You can imagine that we thought it would be a good idea to launch a preList about COVID preprints. And that was incredibly popular initially. But you will not be surprised to hear that we immediately lost control of the number of preprints being posted.
CLAIRE MOULTON: So a few of the preLighters actually moved to produce something separate from our site. And that is a site called covidpreprints.com. And we thought this was quite interesting because it presents the preprint alongside a timeline of the pandemic. And crucially, it calls out preprints that have been debunked, either because they overstated their claims, were too preliminary, or were potentially fraudulent.
CLAIRE MOULTON: So that's actually been a really interesting separate initiative. And I thought just to finish, I would highlight a new service coming out of eLife that they have just called Sciety. And this will be a gathering together of feeds from a number of services that either highlight or review preprints. So I think that that's one to keep an eye on.
CLAIRE MOULTON: So there we are. That's me finished. Thank you very much.
WILL SCHWEITZER: Thank you, Claire. Tasha?
TASHA MELLINS-COHEN: Apologies. Let me just unmute myself. So Will referenced that earlier this year, the Wellcome Trust and Howard Hughes Medical Institute launched the Learned Society Curation Awards. And it was very exciting for the Microbiology Society that we were successful in our application to convert one of the Society's journals, which is Access Microbiology, into an open research platform. And this along the lines of Faculty of 1,000 or Wellcome Open Research.
TASHA MELLINS-COHEN: From a values perspective, this fits very well with the Society. It offers us an opportunity to bring transparency to a traditionally very opaque process of peer review, and embrace open scholarship, which is very much the direction that we see publishing traveling in. It also helps us speed up the process of disseminating research, which maximizes the speed of communication as well as the potential impact and influence of each article.
TASHA MELLINS-COHEN: We've just heard from Claire about preLights, and how valuable that is proving to be as a way to communicate research at an early stage. The reason that we are thinking of this as anti-curation, as it were, is that a traditional journal was hyper-selective. They had to confine the published research to the bound contents of a print journal. Modern journals have somewhat broken those boundaries.
TASHA MELLINS-COHEN: And we now have many journals, many articles which incorporate open data, open methodology, so on and so forth. But we see an open research platform as a new form of product. It's not just a journal. It gives us an opportunity, for example, to incorporate posters from our conferences, to potentially publish the theses of our members if they wish to do so.
TASHA MELLINS-COHEN: There's so many more things that can be incorporated into an open research platform that it becomes less about curation and more about layering and anti-curation. So as you can see here, this is a very simplified, but very traditional workflow for a journal, well, traditional in the sense of preprints having come into play about 20 years ago.
TASHA MELLINS-COHEN: Everyone could see a preprint. We know that, we appreciate that. But from the point of preprint to the point of formal publication is very opaque. A technical check will happen, but if an editor decides at that point to reject the article, nobody sees that information. Nobody sees why that rejection has happened. Similarly, peer review, nobody really sees the peer review reports except for the editors and the authors themselves.
TASHA MELLINS-COHEN: You don't have that level of transparency. And only after many rounds of review and revision and editing, and is a very highly curated process, will the real world, the world outside of the editors and the reviewers and the authors, see the final version of the article. So most research, most of the process of research is curated, and very carefully so. For an open research platform, the workflow changes.
TASHA MELLINS-COHEN: And it looks a little bit more like this. So there are many, many points where versions and reviews of those versions are posted. So our intention is that the point an article is submitted-- It might well have already gone onto bioRxiv. That's entirely up to the authors-- we will run machine learning technology peer review. So I suppose not truly a peer because it's not human. But we will be looking at things like SciScore and authenticate to make sure that the research meets certain basic standards.
TASHA MELLINS-COHEN: The initial preprint will then be posted to our platform. And that's our standard microbiologyresearch.org platform with the AI review reports made available alongside it. Transparent peer review will be managed through our editorial manager system. We're not making use of any kind of bespoke software here. It's all the stuff that we already have in our technology stack.
TASHA MELLINS-COHEN: And again, the reviews will be posted once they've been received. And the revised article will be submitted. And we'll see a new version of the preprint, and so on and so forth. Everything will include relevant persistent identifiers, whether that's DOI for all of the versions of the article and for all of the peer review reports, links to open data, use of funding data, usage of Research Organization Registry information, use of ORCID, all of that will be incorporated at every point in the process.
TASHA MELLINS-COHEN: So in this way, we get a very clear very transparent, very versioned view of the research process. Now, this bit is anti-curation. What comes next is a bit different. After this, with this formal publication of the the version that the reviewers and the editors and the authors agree is acceptable, the communities around the Microbiology Society and, indeed, beyond the microbiology society, will have an opportunity to create, effectively, overlay journals or reinvented curation, but reinvented curation over the top of a complete transparent process.
TASHA MELLINS-COHEN: I thin k that is it. Yes. I will hand over to Bert.
WILL SCHWEITZER: Thank you very much.
BERT CARELLI: Excellent. Thanks, Tasha. Thanks, Will. Let me just. So TrendMD is a technology company. So I'm going to focus a bit on technology. And to provide a bit of oversimplified context here, we can look at technology as having played an evolving and disruptive role in the curation of scholarly content.
BERT CARELLI: Up until the 1960s, curation was the province of editors in libraries. Manuscripts were selected in issues assembled by editors, in libraries acquired and organized by journal title, volume, and issue, discovering new advances in one's own field, and first acquiring access to the key journals, and then systematically browsing tables of contents, or in a few fields, like medicine, browsing scholarly indexes that could be months or years out of date.
BERT CARELLI: These methods for keeping up with new discoveries were highly inefficient, and often prohibitively expensive. NASA learned during the early days of the space race that it was often more expensive to figure out if an experiment had already been done than it was to simply do it again, from scratch. In fact, that realization led to the creation and funding of one of the first information retrieval databases at Lockheed Missiles and Space, and the era of digital creation curation was launched.
BERT CARELLI: Database era ushered in major changes in the way scholarly information could be curated. Instead of pouring through journal by journal, issue by issue, a researcher, usually with the help of a librarian skilled in the specialized language of command line search, could look at the entire archive of a journal, or search a citation database using subject taxonomies like MeSH, and curate lists of appropriate articles for the researcher to review.
BERT CARELLI: Flashing forward to the first two decades of the 21st century, web technology has given us the ubiquitous search box, from which virtually all scholarly material can be accessed, and where curation is effectively done at the article level. A keyword search on any subject instantly "curates"-- I'm using air quotes here-- a list.
BERT CARELLI: The quality of that list can vary widely. For example, you might see a peer reviewed article from an established journal on a subject like the flu vaccine sitting next to a conspiracy monger's post claiming vaccines can cause autism. But accepting that the serious researcher can deal with its obvious flaws, we can see that curation, the Google search does fulfill another purpose. Sometimes you're not just looking for a specific piece of content, which Google's obviously very good at providing.
BERT CARELLI: Sometimes a researcher also wants to see what other related content might be interesting or useful. In this case, the search results page becomes not the end, but a key step in the larger process of discovery. In their survey, "How Readers Discover Content in Scholarly Publications," Tracy Gardner and Simon Inger found that users are also likely to browse other articles that surface in a search.
BERT CARELLI: So the need to browse to learn what you did not think to ask, maybe allowing for a little serendipity, is still a vital part of the research process. And this is where this related articles function comes into play, which is TrendMD's business. Going back to the study that I just mentioned, which has been done every three years since 2005, users were asked what features of published websites they use most often.
BERT CARELLI: Over the five iterations of that survey, the related articles feature showed consistently increasing popularity, while other website features, like publisher produced news, site search, saved search, alerting, declined. TrendMD's recommendations widget uses behavioral data and AI technology to create another type of browsing experience for the reader, suggesting additional articles based on what they are reading at the moment, what other users like them have read, and what they've read in the past.
BERT CARELLI: Unlike a search-generated browse list, which contains thousands of items closely related to the search terms, TrendMD's focus is to curate a short list of related items with the highest likelihood of being of interest to the reader. It's similar to the way that Amazon works. If you buy a teapot through Amazon, the site's not going to show you more teapots.
BERT CARELLI: You'll see products that other buyers of teapots have bought, like recipe books, teacups, or tea. In TrendMD's case, a medical practitioner reading an article in The BMJ on smoking hazards, may see recommended articles on lung cancer. While a public health policy specialist may see articles on secondhand smoke, and a biochemical researcher may see articles on tumor formation. The other aspect of TrendMD's approach is that these recommendations are selected from the largest possible collection of content.
BERT CARELLI: Half of the recommendations are from the same journal or publisher as the article being read. And the other half are from a network of over 5,000 scholarly sites. Result is that the user sees content from journals and publishers that they might otherwise not be familiar with. And it enables authors and publishers to reach audiences that will truly value their content. Number of case studies, including a 12-month randomized control trial that demonstrated that articles promoted by TrendMD were 50% more likely to be cited, have shown the value that this feature provides for publishers, authors, and researchers.
BERT CARELLI: Back to Will.
WILL SCHWEITZER: Thank you very much. So just want to remind the audience, you all can submit questions via the Q&A, which you'll find at the bottom of your Zoom screen. And we have just a few questions to kick off our discussion. And Claire, I thought I'd come back to you first. And you've already answered two audience questions around who are the preLighters, and how were they selected and trained. But do you know if the preLighters are using any technology to either identify, or select, or even summarize preprints they may find?
WILL SCHWEITZER: Is that something you all are looking into?
CLAIRE MOULTON: No, it's not actually. So I think it's a really good question. But is a very manual human process at the moment. And I think that we did have in mind that we might try those sorts of things in the future. But I think people quite like the real personal touch. And one of the things that has really enthused people about this product is actually the early career community. And they are really keen on preprints.
CLAIRE MOULTON: They write preprints with meta-analysis about preprints. And they do lots of presentations about preprints. And so it's very much a human endeavor at the moment.
WILL SCHWEITZER: Thank you. So Tasha, I'm going to purposefully ask a pointed question here to maybe generate some more discussion? And, actually, I have two questions. And one is, do you feel this ant-curation approach exacerbates the glut of information that's out there?
TASHA MELLINS-COHEN: I think the flood of information is happening anyway. The anti-curation approach, by introducing transparency and providing levels of reporting even on that very first preprint, which simply doesn't exist in bioRxiv, helps readers to understand that what they're seeing might not be the final version, but has been through some level of analysis, some level of checking.
TASHA MELLINS-COHEN: Interestingly, when Claire mentioned early career researchers, Access Microbiology as a journal was originally created specifically in response to Early Career Microbiology Forum saying that they needed a place to publish small papers, replication studies, negative results, and so forth. And actually, not only was that journal's scope defined by the early career group, but the editorial board is made up primarily of early career mentees, paired with very experienced mentors.
TASHA MELLINS-COHEN: And that group has proven to be very receptive to different ways of working, including this kind of anti-curation approach. So it was it was them. It was the early career group who were really instrumental in us applying for the Wellcome Grant. And they're the ones who are saying, this will make our research easier, better because we will be able to see what is valid immediately.
WILL SCHWEITZER: Oh that's great. And then I know it's kind of relatively early days for the journal over the open resource platform. But are you seeing any user data that suggest readers are accessing and using either the peer review commentary, the decision letters, or the material around the final version record article?
TASHA MELLINS-COHEN: So we aren't live yet. We're still very much in build. It's very early days. But part of the reason that we went ahead with this is that, actually, if you look F1,000 or Wellcome Open Research, you can see how much traction the reviewer reports are getting on those platforms. So you can see how many times those have been read. Now with any download statistic, any usage statistic, you don't know for sure if somebody's actually read the piece, or if they've just opened it.
TASHA MELLINS-COHEN: But we know that people are making use of that. They are visiting the peer review pages, which gave us confidence that if we were to do this, we would be able to add value to the publication process by adding transparency.
WILL SCHWEITZER: That's excellent. I think it's been, I would say, an increasing trend we've within this Silverchair community that publishers are bringing new questions to their platform analytics and their usage data.
TASHA MELLINS-COHEN: Yeah.
WILL SCHWEITZER: It seems like five years ago, we were all taking for granted that folks were coming in from Google or a large-scale search engine, finding the PDF that they wanted, and leaving. And I think as we dig in further, we're finding there's really valuable use cases and services we can add on.
TASHA MELLINS-COHEN: Yeah.
WILL SCHWEITZER: Bert, my next question is for you, which is you provided some great data around how frequently the TrendMD widget and recommendations are used on platform, some potential citation effects for the widget, which is really great data. But I'm thinking about editorial stakeholders in the process, and the efficacy of the recommendations, and if you have any data, anecdotal or otherwise, that's compares, say, TrendMD's recommendations versus editorial or human-curated links in the process.
BERT CARELLI: Well, yeah. Thanks, Will, for that question. At your former organization, we went through a very lengthy trial, starting out on some of the newer journals, the smaller journals. And the editors weighed in heavily in the early days about the TrendMD recommendations. And the conclusion, and why we're now on all of the AAAS journals, was that they actually preferred the engagement that was being shown for the TrendMD recommendations.
BERT CARELLI: And it's actually not surprising because from my experience back working with publishers over the years, I've noticed that publishers are not so attentive to the use of features on their website. And one of the things that has been on websites for many years is some type of related articles functionality. And often, it was one of the least used functions even when editorial staff spent a lot of hours trying to curate that list of, here's additional content that the user might be interested in.
BERT CARELLI: So being able to come up with it, being able to offer an automated solution was helpful, certainly from a cost perspective for the print publishers. But also, the engagement that was provided clearly indicated that people were using them. And that's really what TrendMD functions on is clickthrough rate as the key indicator of quality. If people are engaging with the widget, if they're clicking on the articles, then it's a pretty good indication that there must be some quality there.
BERT CARELLI: So it's really leaving it up to the users, the wisdom of crowds, if you want to think of it that way, to dictate whether the quality is there. And so far, it's been consistently high.
WILL SCHWEITZER: Thank you. So I have one last question before you all if you have questions for each other. And then we can turn to some audience questions, which is we've brought up kind of COVID. And we've seen a proliferation of activity around preprints and COVID. Chuckling at this point may be a good coping or safety mechanism. But earlier today, Kent Anderson in The Geyser wrote about the potential problems of preprint servers, and essentially disinformation.
WILL SCHWEITZER: And I think it points this really interesting tension of the role of curation, and essentially signaling significant and valid research. And if I were to put an editorial comment on preLights, or define communities that I see, it seems to be working with a closed community. So it may be working within a group of several thousand life sciences researchers, but we don't seem to have effective curation mechanisms, yet, at scale, or at the seam of science and the broader public, or, perhaps, even science in the policymaking community.
WILL SCHWEITZER: I was wondering if you all have any thoughts about this. Are you seeing anything in your communities with your programs that might give possible solutions or indications for solutions for this problem about curating research or even early stage research for the problems facing us today?
TASHA MELLINS-COHEN: May I?
WILL SCHWEITZER: Please.
TASHA MELLINS-COHEN: I would argue that there are also problems with research that is published in a journal post peer review. One of the biggest public health crises we faced-- this would be prior to COVID-- is the damage done by the Wakefield paper, which was published after peer review in a very well-respected journal. And it has created a hideous situation, where many people are rejecting vaccination, which is a perfectly safe and sensible public health measure.
TASHA MELLINS-COHEN: And it's allowing terrible diseases to spread again in communities that should not have to face it. So, yes, I absolutely agree that the proliferation of non-peer-reviewed research being available to everybody could create a problem with disinformation I don't think the peer review is necessarily the only way to solve it. But I do think that the application of the sorts of machine learning tools that we are proposing to use for preprints could help to flag, for example, there is a methodology problem here, or these authors have not declared their conflicts of interest, and give people some context and some guidance as to where there may be flaws in that manuscript.
TASHA MELLINS-COHEN: But we have for decades said, peer review means something is valid. And that is not necessarily the case. And Kent and I have had a vocal argument about that, so [LAUGHS] I won't carry on.
CLAIRE MOULTON: I would agree with that. I think that posting or publishing, wherever if you are in the cycle, it's still just part of the beginning of the discussion of the science. People want to discuss it, they want to reproduce it, and they to challenge it. I think that one of the interesting things that is going to come out of this past year is that I think that the people who were in favor of preprints will be feel even more strongly about it because they can see that the rapid fabrication of work has played a very important role this year around COVID.
CLAIRE MOULTON: And the people who have had concerns about preprints will feel more strongly about that because they will have seen preprints that could potentially cause harm or, certainly, diverted resources. So I suspect that the viewpoints may have become even more opposite.
WILL SCHWEITZER: Seemingly with all things these days. Bert, any thoughts from you?
BERT CARELLI: Well, I think that there's a certain philosophical perspective here. When Cold Spring Harbor first launched bioRxiv, I was working with them to help them launch the preprint site. And, to be honest, I had some skepticism after working with scholarly journals, with the peer review journals for so many years, and knowing how editors were so protective of their space, of their choices, of what to publish.
BERT CARELLI: But I became a convert, to be honest, especially seeing the growth and the enthusiasm. I have a son, who at the time was in graduate school and working in biochemistry, and I often used his group, his peers to show me early examples, even before bioRxiv launched. This is the type of thing-- This is what is being attempted here. And the overall enthusiasm of those young people, of those early career scientists, who are now know actually in there the middle of their careers, was overwhelming.
BERT CARELLI: And I think that there's a sense that if we can trust the process-- There are going to be these outliers, there are going to be the Wakefield papers, and there are going to be some of the COVID papers that have been really scandalous. But there's a certain wisdom in crowds that we're almost committed to that today anyway because we can't deny that the tides are rising around us all the time of information.
BERT CARELLI: So we have to trust that there's an audience there that has the ability to make some critical choices about things. So, I don't know, that that's my sense in why I feel like a technology solution, you can't turn the clock on it. Once the genie is out of the bottle-- to mix many metaphors here.
WILL SCHWEITZER: Still a trust or hope for audience discernment. So before turning to some audience questions, we have three in so far, and some of them are a bit philosophical, I wanted to see if you all had any questions for each other.
CLAIRE MOULTON: I do have a question for Tasha, actually. It's quite a practical one. But when I was talking about your new workflow with my team, they were interested in, what happens if a preprint comes to you, goes through the review, and then the author chooses not to do the revisions that you've requested? Can they then continue and take that paper elsewhere? How does it work?
TASHA MELLINS-COHEN: Well, effectively, it is a preprint. So if they really don't want to continue, then, yes, they can take that elsewhere. And they can say, this is my preprint, it's in the open research platform, I have chosen not to finish the publication process there. There's no reason that we couldn't let that happen. From the conversations that we've had so far, the researchers that we've spoken to have sort of said, oh, why would I do that?
TASHA MELLINS-COHEN: If I've gone through all that process, and I've gone through the reviews with you, why would I take my article elsewhere and start again? So I don't foresee it necessarily happening particularly frequently, unless and until there is a pretty broad acceptance of transfer of peer reviews, which isn't hugely popular at the moment. And there's a lot of editors that I work with personally who've over the years said, I wouldn't trust somebody else's peer reviewers.
TASHA MELLINS-COHEN: Why? If the peer reviewer's good, the peer reviewer's good. So, yeah. Hypothetically, people could get partway through the process and pull out at that point. There's also a possibility that they get partway through the process and refuse to meet the requirements of the editor to amend their paper to make it satisfactory.
TASHA MELLINS-COHEN: And at that point, it would remain simply a posted preprint. We wouldn't go through the production process, which I think is possibly more likely to happen.
WILL SCHWEITZER: Any other questions for each other?
BERT CARELLI: Yeah, actually I have a question really probably more for Claire, but also Tasha. I've been talking with a number of companies that work with automated summarization and sentiment analysis, and I'm curious whether you've discussed or considered adding this type of thing to somehow scale up the process? Especially, Claire, it's impressive the numbers that you mentioned of the preLights involvement.
BERT CARELLI: But still, it's a very small fraction of the preprints that are there out there.
CLAIRE MOULTON: As I say, it's a really good point. It is one that we have discussed. I think that we're less likely to do it on preLights because I think that one of the parts of the post is the, why did I personally choose this preprint? And what are the questions that I want to ask to the print author? And be able to follow up in more detail with those. I think that where we more interested is-- So often, with research articles, we write a research highlight, so just a short summary the basically encourages readers to go and read the article.
CLAIRE MOULTON: And that's the sort of work that our in-house team currently write. And I can see a time saving for using an automated process for that. And we would still have I think that certainly in the early days, would, certainly then have a person checking that before it went live. But, yeah, we are interested in the idea.
TASHA MELLINS-COHEN: So we are actually going to be using one of those tools as part of our initial pass, initial automated pass through each preprint, so that we can give people an idea of what's in there. Because why not?
WILL SCHWEITZER: Great. It'll be exciting to see how that gets on. This next question's from the audience. And it's a bit of a philosophical one, which is, is there a distinction between recommendation and curation? There's probably a related point here about how do we label those recommendations or curated links for end users.
WILL SCHWEITZER:
TASHA MELLINS-COHEN: I think I would draw a distinction because there are articles that I might collect together for a particular purpose, and I would consider that to be curation. But the purpose might not be positive. It might not be something I would recommend that somebody read. I might be collecting it together to debunk it or to another way challenge it. I don't know how the others feel about that.
CLAIRE MOULTON: Yeah, I would also distinguish between them. I think, yes, curation is more the collecting together than a recommendation. But I suspect the most curated collections will be recommendations. But I think they are different.
TASHA MELLINS-COHEN: Yeah, and a reference list is a curated list. But it's often, this is fake, this is negative. Sorry, Bert.
BERT CARELLI: Yeah, well TrendMD comes at this issue, and it's a big question for us, certainly, that we think about all the time from really a slightly different perspective. The problem that we set out to solve was really one more for publishers and authors who wanted to have their content exposed to broader audiences. So the purpose was somewhat different. And I really appreciate, Tasha, your mentioning curation for other kinds of uses.
BERT CARELLI: So we're really talking about really different intent more than the difference in what the feature actually just throws out there on a site. And I think, probably, there could be more clarity around that from the perspective of labeling these lists of recommendations as recommendations that are coming from a particular perspective, that these are what the author and publisher wants you to see, which is what you're going to expect to find on a journal site.
BERT CARELLI: These are the issues that have emerged out of the content, maybe, perhaps, but from a different perspective, from the more qualitative analysis of the site. That's kind of where I was coming from about the question about automated summarization and sentiment analysis. I wonder if there are multiple opportunities for recommendation technologies that could be applied in these different ways.
CLAIRE MOULTON: And I think recommendation doesn't always mean that you agree with the finding. It could be that you have back to back papers argue two sides of an argument. But you would recommend that people read both of them to see the two sides of the argument.
WILL SCHWEITZER: That's a really good point, great. So this is a related question, and it is, how do the goals of users and the goals of publishers come into play in curation? And do those goals conflict at times?
CLAIRE MOULTON: I think that when you are launching something like this, you're aiming it at the user. Of course you want to, perhaps, promote the fact that you have some strong content in an area, or something like that. Or you want to present some content because you want more submissions in that area. But you are always thinking about the user. That's why you produce it.
CLAIRE MOULTON: If people aren't reading it, then it hasn't succeeded. I'm not sure that I would say that aims conflict. I'm sure that publishers get it wrong sometimes, and produce a collection that the readers aren't as interested in as we'd hoped. For example, I think that when they announce the Nobel Prize winners, lots of publishers suddenly go, oh, yes, we've got some papers on that! And we've got a fantastic paper by that particular author.
CLAIRE MOULTON: And we pool collections together, and then not always very successful. We feel that we need to showcase the work. But we do at least always have the user in mind.
TASHA MELLINS-COHEN: I think, as well, the idea of "the user" is really interesting to me because we have many different communities we are trying to serve simultaneously. So if you're a publisher, you need to serve authors, reviewers, editors, and librarians, and funders. If you're a society, you also need to think about your members, and the different types of members. So the idea of a user, I think, generally, we're trying to serve a specific group of users, not necessarily everybody, all the time.
TASHA MELLINS-COHEN: So, yes, I'm sure there are times when a publisher is launching a new journal, and a library is just going, god, we don't need another journal. Please not another journal. That's just going to cost more money. But the editors, the people who want to launch the journal, are saying, there is a niche and nobody is publishing in this area.
TASHA MELLINS-COHEN: So, in that instance, we are curating content as a publisher with conflicting demands from the two different types of user.
BERT CARELLI: And users are not necessarily monolithic in their process. When you think about, well, again, coming from the perspective of TrendMD, where we find the most usefulness, where users really raise their hands and say, yeah, this was really useful to us, is when they are looking for that piece of serendipity. They're looking for that. One of the other topics that I may have missed, so it's, if you think of it as an early part of the research process, maybe it's the PhD student who is fishing around, like, where am I going to focus the next few years of my career?
BERT CARELLI: And it's also, to some degree, the person that runs a lab is looking, like, what are the experiments that I should be planning for in the next two years? So it's more keeping the wide end of the funnel as open as possible. Whereas, you know you could really imagine curation at a later stage, where you really need to be conscious of, is someone else publishing a paper that's close to this particular topic?
BERT CARELLI: Are these compounds being used in another way that I should be aware of before I publish? So I think it plays a role in all those different stages. So it's very much a moving target, is what the user really is interested in.
CLAIRE MOULTON: I think you're spot on there, actually. I think that it's finding the balance between wide and narrow. And, certainly, the feedback that we get from researchers is that whether a list is manually curated or automated, they worry that it's not going to include the one thing that they would have filed if they put the list together. But at the same time, if you always do a broad list, then they're not going to find all of the pieces quite so interesting.
CLAIRE MOULTON: And it is a really crucial part of the puzzle.
WILL SCHWEITZER: This question is from our friend Richard Wynne, which is, does curation include establishing the specific contributions of each author thinking of credit attribution?
TASHA MELLINS-COHEN: It could. It doesn't always. In practice, in reality, I think it would be a wonderful thing to be able to say, yes, every author on this paper has a credit tag. But it doesn't always happen.
WILL SCHWEITZER: And I guess that brings up a slightly broader point, which is, how should we be valuing or recognizing the act of curation? Is that a similar level of importance, say, as authorship? I think as the credit taxonomy is designed to recognize. Any thoughts on that?
BERT CARELLI: Well, it comes back to the value of libraries, I think, at its core. The library was always responsible for curation. And the value of librarians themselves and their expertise to field. The advantage that a librarian has is of impartiality. Their only skin in the game is to serve the user, the researcher.
BERT CARELLI: So I think that that's where the value ultimately should be. That's the attention that we should always be putting on this whole process.
TASHA MELLINS-COHEN: There's also a question in there about recognizing review and editorial contributions because, at the moment, again, most of that is hidden, mostly it's not named. Even if you're providing ORCID feeds with information about who's contributed to peer review, It's not really clear. And that is a part of the curation process. That editorial selection and peer review is a part of the curation process.
TASHA MELLINS-COHEN: And we don't really have a proper taxonomy around that. So if somebody has done, for example, a statistical review or a methods review versus a general peer review, we have no really clear way, within CRediT, or any other taxonomy, to define that. And we don't describe our editorial processes, particularly in a standardized way. So there is a question there about the role of taxonomy for the process of curation, as well as for credit for the authors.
TASHA MELLINS-COHEN:
WILL SCHWEITZER: And I think for our last audience question, Claire, this one is for you. And I'm trying to synthesize on the fly here. But it seems like a company biologist is making it an investment either in human time or dollars in the summaries. And then whether you have thought about repackaging those summaries for audience, and whether that would have revenue-generating potential.
CLAIRE MOULTON: So we did decide to invest in this area. And we don't currently have any commercial revenue-generating ideas at all. We've specifically set those aside, so that we could just develop the project, and take ideas from the preLighters themselves, and invest in those, as well. So we've done quite a few of those sorts of things. I think it depends on the nature of the business. We have tried to set ourselves as a community publisher.
CLAIRE MOULTON: We have a number of community sites. We have one called the Node, which is for developmental biologists, so it's associated with a particular journal. And we have one which we launched just this year, called focal plane, which is about imaging and microscopy. And that relates to all of our journals in some way. And we've really just invested in those. They don't cost a huge amount of money.
CLAIRE MOULTON: And we feel that they do provide a big benefit to the community. And so we decided it was part of our-- We're not a society, but we are a community publisher. And we just thought it was part of our community service to provide those forums.
WILL SCHWEITZER: And there's a second part to this question, which is, do you have any comparative data, say, comparing how these summaries are used or how preLight is used relative to, say, your original peer-reviewed research and in your journals?
CLAIRE MOULTON: Well, I don't think we've really tried to compare them as such. Clearly, they have a very big Twitter presence. All journals have a Twitter presence, as well. But this is especially hot on Twitter. And I think that they will be the full piece will be read fewer times than one of our published articles because I'm sure that a lot of people read that the tweetable summary, and they always feel like they read the article.
CLAIRE MOULTON: A little bit like when someone spots an article on a table of contents, they think they've read it. But I think that it does create an excitement and a visibility for the preLighters and what they've done. A lot of people tell me that they use preLights a lot. And so you know I think that the warm feeling that people have is one of the benefits of having produced this service.
CLAIRE MOULTON: And one of the reasons-- Obviously, there's always a little bit of a selfish reason for launching a product-- But we did feel that preprints were hot. We wanted to show that our journals were preprint friendly, we accept reprints, we do transparent peer review, we do a lot of things around transparency and openness. And we thought that this would, in some ways, reflect on us as an organization, that we'd embraced preprints and early career researchers.
CLAIRE MOULTON: And that has definitely paid off. So if you were looking for sort of a selfish nugget in what is really a community initiative, that would be it. I think it made us look cool.
WILL SCHWEITZER: Thank you. Well, do you have any last questions for each other?
TASHA MELLINS-COHEN: Not for me.
CLAIRE MOULTON: No, good discussion.
WILL SCHWEITZER: Well, first, a very big thanks to you, Claire, and Tasha, and Bert, for joining our panel today. And a really big thanks to our audience for joining us and participating. As my colleague, Stephanie, mentioned at the beginning of the webinar, registrants and attendees will receive an email with a link to this recording, and a link to a participant survey. Your feedback will be really helpful to us as we plan for 2021.
WILL SCHWEITZER: So please, take a moment to complete it. I'd really appreciate it. So thank you all again. And we'll see you on a webinar, or perhaps, even at a live event sometime soon.
BERT CARELLI: [LAUGHS] Let's hope so.
TASHA MELLINS-COHEN: Thanks and goodbye.
CLAIRE MOULTON: Thank you, bye.
BERT CARELLI: Thank you both. Bye, Claire. Bye, Tasha.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode