Cadmore media player playing video What’s the Issue? Impact and Metrics
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
What’s the Issue? Impact and Metrics
Description:
What’s the Issue? Impact and Metrics
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/096cb9d9-d536-4bf5-9a10-e4af0ca50800/thumbnails/096cb9d9-d536-4bf5-9a10-e4af0ca50800.jpeg
Duration:
T00H59M17S
Embed URL:
https://stream.cadmore.media/player/096cb9d9-d536-4bf5-9a10-e4af0ca50800
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/096cb9d9-d536-4bf5-9a10-e4af0ca50800/What%27s the Issue.mp4?sv=2019-02-02&sr=c&sig=nM7151A3UJxbMTcKw%2FYb4T%2F7o%2FkzbSbuif1YL116O%2Fw%3D&st=2024-11-20T00%3A36%3A13Z&se=2024-11-20T02%3A41%3A13Z&sp=r
Upload Date:
2024-02-02T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
MARY BETH BARILLA: Hi, everyone. Welcome back to day 2 of the SSP New Directions seminar. If you were not able to join us yesterday, my name is Mary Beth Barilla and I'm the program director at SSP. I want to thank and acknowledge our sponsors again today. Access Innovations, Atypon, Cadmore Media, eJournal Press, Impact Journals, Jack Farrell & Associates, Silverchair, Straive, and Taylor & Francis, F1000.
MARY BETH BARILLA: Thanks so much for your support. We've got some housekeeping issues for you. At any time during today's sessions, if you need support, life support is available. Just go to that Support tab there at the upper right in Pathable to find a link and it'll connect you with a person. If all else fails, customer service's email is info@sspnet.org. We will enable closed captions for our sessions today, and you should be able to access that CC button on your Zoom toolbar.
MARY BETH BARILLA: We are also recording today. And if you miss part of a session or want to come back and listen to a particularly interesting session, you'll be able to do that. We'll post those at the end of the seminar. And we do want to hear from you again today. So if you Tweet, our hashtag is #sspND2021. There's a Q&A box to direct questions directly to the panelists.
MARY BETH BARILLA: There's also a chat feature. Be sure in the Zoom chat to select everyone if you want to talk to all of your fellow attendees, or just select hosts and panelists if you prefer. But just be mindful, you have two choices there. And now I'd like to turn things over to Sophie Reisz, Working Group Lead for our New Directions Working Group.
SOPHIE REISZ: Thank you very much, Mary Beth. Hi, everyone. I'm Sophie Reisz, Vice President and Executive Editor at Mary Ann Liebert, Inc Publishers. It is my great pleasure to introduce our first session of the day, What's the Issue? Impact and Metrics, moderated by Dr. Sai Konda. Sai is the Senior Managing Editor in the Global Journal's Development Publications Division at the American Chemical Society, and is also an SSP board member.
SOPHIE REISZ: Sai, over to you.
SAI KONDA: Thank you, Sophie. Warm welcome to everyone. Welcome to day two of the New Directions seminar. Just before we get started with introductions from our panel of speakers, I wondered also give a shout out to Alexa Colella from University of Illinois Press for getting the session together and putting the panel together. So thank you, Alexia, in addition to all the members working behind the scenes. So just to get started off with the session, I wanted to get some introductions from our panel of speakers.
SAI KONDA: If you can briefly introduce yourselves and talk about your roles in your respective organizations. Maybe we can start off with Josh first.
JOSH NICHOLSON: Great. Thanks for having me, and thanks, everyone, for tuning in. My name is Josh Nicholson and I'm the co-founder and CEO of scite. I have a background in cell biology. So I finished a PhD about six years ago. I have always been interested in kind of how science is done, which is, I think, led to where I am today. And I feel like I have this kind of love-hate relationship with metrics and citations where I kind of hated how they were and how they're used in some cases so much that I've tried to do something about them with scite.
JOSH NICHOLSON: And I think now, used in the right way, they can be incredibly powerful, and so I'm excited to talk about our use case at scite and what we're doing. Yeah.
SAI KONDA: Thank you, Josh. Rebecca, next.
REBECCA KENNISON: Hi, everyone. I'm Rebecca Kennison. I'm the Executive Director of K|N Consultants, which is a non-profit agency that works with higher education institutions and organizations, including academic publishers and libraries for all the librarians, to enact transformational change. And then my background is both in publishing. I was an academic publishing for a very long time, including open access publishing.
REBECCA KENNISON: So yesterday's sessions were really, really interesting in that regard. And also, I worked in an academic library for eight years. So I think I understand both those sides very well. There are a couple of projects that are going to help to inform what I'm going to-- some of my comments, whatever my comments might be later. Both projects were generously funded by the Andrew W. Mellon Foundation.
REBECCA KENNISON: But the first one was publication ethics, which I worked on with several philosophers and with the executive director of the American Philosophical Association. And in particular, that project, while it looked at publication ethics of all kinds, it was very interested in bias and citation practice. And so in the humanities, and for people who know philosophy, philosophy is the straightest, whitest, oldest of the humanities.
REBECCA KENNISON: And so it was a really interesting project. And it did really transform my thinking in how I look at citations and think about citations. And so there's that one. The second project that I'm currently involved in is the Humane Metrics in Humanities and Social Science Initiative or Humetrics where I'm a co-PI. And among other outcomes, we're looking to shift thinking in the academy so that we focus on measuring what we value rather than valuing only what we can measure.
REBECCA KENNISON: And so there will be a lot of that aspect that I also bring into my comments today. Thanks.
SAI KONDA: Thank you, Rebecca. And last but not the least, Marie.
MARIE MCVEIGH: I was hoping you would say that so I wouldn't have to or so that I wouldn't feel least. I really appreciate being part of a panel where in some sense I do feel least, because I really respect these two individuals and have followed their work across years. Background, I joined Clarivate in 2017 as part of the development team for the Journal Citation Reports.
MARIE MCVEIGH: I wanted to bring a new sense of that dynamic relationship between article performance and journal performance. And to open up a much greater transparency on the components of the journal impact factor, because that had been such a matter of discussion and debate and dismay. In 2019, I moved to the editorial team to continue what has been a career-long interest in bibliographic and bibliometric research.
MARIE MCVEIGH: And to work in the integrity and maintenance of the scholarly record that is reflected in the Web of Science Journal resources. Before Clarivate, I worked in a bunch of roles in primary publishing, in A&I analytics, and author services, in editorial support systems, but I really started this whole adventure at the Institute for Scientific Information as the Selections Editor for the Life Sciences Content, coming out of a cell biology degree, I will say.
MARIE MCVEIGH: I was there for the first launch of the Web of Science. I've also, for about 10 years, I led the production and product development to the Journal Citation Reports prior to its new installation. And I've also-- I started up there doing surveillance and review of citation metrics with the manipulation of metrics, with the use of detection on the misplaced priorities that researchers bring to try to juke that system.
MARIE MCVEIGH: At ISI, I had the privilege to study under Dr. Garfield. And I continue some research into bibliometrics and ethics and publishing and citation metrics. And the real value of citation indexing, I'll say, I let Garfield's original intentions for citation indexing be kind of a lodestar for me. I'm still an idealist all these years later. And what is really important underneath all of that is that this is built on a fundamental aspect of the scholarly dialogue.
MARIE MCVEIGH: And I think we all agree that that itself is a point of value.
SAI KONDA: Thank you so much, Marie. It's great. I think it's already evident given the diverse backgrounds and experience of our fantastic panel of speakers. So before I jump into my open-ended questions to the panelists, just to remind the audience that please feel free to use the Q&A box in Zoom to forward your questions. We will try to take them during the discussion as humanly possible. If not, we'll try to have a Q&A session at the very end so that we can have a discussion on different topics.
SAI KONDA: So let me leverage all your diverse experiences and start off by asking you what your perspectives on where metrics stand and why do you think metrics are important? Let's start off with Marie.
MARIE MCVEIGH: OK, first and least. I'd say that metrics in their best instance provide a verifiable checkpoint. And I'm talking about metrics generally. We'll touch citations throughout, I'm sure. Verifiable and transparent doesn't make a metric perfect, and it doesn't make it fit for every purpose, but it can create a shared fact base. Which is why I like metrics, it's a place to begin to put things on the table and say, this is what we are looking at and this is what we can begin discussion with.
MARIE MCVEIGH: Like any data, metrics are not a substitute for judgment, they're a tool to guide and share judgment and to share that sense of fact base. They are support and an opportunity to generalize the principles that underlie a decision. And I want to put like right upfront why do I think metrics are important. I want to put upfront at the opening of the Leiden Manifesto, which was-- I visit again quite regularly, research evaluations that were once bespoke and performed by peers are now routine and reliant on metrics.
MARIE MCVEIGH: The problem is that evaluation is now led by the data rather than by judgment. Metrics have proliferated. Usually well-intentioned, not always well-informed, often ill-applied. We risk damaging the system with the very tools designed to improve it.
SAI KONDA: Thank you, Marie. Rebecca, next.
REBECCA KENNISON: Just to piggyback on what-- very provocative there, Marie, in terms of-- I think what people understand is that metrics-- like everything, including impact factors and so on, is designed to solve for problems that would help this to be a much more equal and equitable environment in which to operate.
REBECCA KENNISON: And what has happened over time, as the Leiden Manifesto points out, is that we've become data-driven instead of data-informed if I could put it that way. And we don't necessarily understand the context within which we're seeing whatever that number might be. And whether that's an h-index or an impact factor or just counting up citations and so on, I mean, as I mentioned, one of the things that I'm really interested in is citation practice.
REBECCA KENNISON: And in the humanities, this is a particularly important thing, because that's all you got. Basically in the humanities is you have a conversation between this piece and now what I think and then people-- now then talk about that piece and so on. This is the way that humanities in particular-- I mean, all of the academic conversations go this way. But this is the sole way that the humanities do their work.
REBECCA KENNISON: But who gets cited, who doesn't get cited? What are the practices that publishers have imposed basically on this? Like limiting citations, which we don't need to do-- we have unlimited space now on the-- those constraints that were [INAUDIBLE] constraints, we should really rethink those, and there were lots of conversations yesterday that I appreciated in the panels among what an online world looks like.
REBECCA KENNISON: But those constraints are really, really critical when what you have to do is choose who you're going to cite. And then-- and this is maybe where Josh will jump in. And then what is the context for those citations I think is really an important thing. And none of that is really captured in the way that metrics are done for the most part. It's not-- the narrative tends to be lost when all you're doing is listing what you're doing rather than why, how, who, and so on.
REBECCA KENNISON: So there needs to be more context.
SAI KONDA: Absolutely. Josh?
JOSH NICHOLSON: Yeah, so I would definitely second that we need more context. And so I think metrics are interesting because there's this tension between them being simplistic and not helping, but then being too simplistic. And so I think you can go away from metrics, but then you'll lose kind of the value of this as a heuristic. And so I think this tension will always kind of exist, and I think we're at an interesting stage where there's a diversity of metrics.
JOSH NICHOLSON: And I worry a little bit that we're getting to having kind of all these flare and badges for every different thing that we're all measuring. So open data for how it's been cited, supporting citations. And I think it could be a good thing, but I think we should be cautious about metrics for everything. I also really like what Rebecca said about the conversation happening amongst papers.
JOSH NICHOLSON: And so at scite, we show the citation context, in-text citation from the citing article, and I've started to really stop saying even citations. Say, look at this conversation that's happening amongst papers. And I think that is also really amazing and kind of where my love for citations is, because here is this conversation across generations, across disciplines that is preserved.
JOSH NICHOLSON: It's backed by data, it's backed by analysis, it's citeable, and I think that is really powerful, because you have this huge knowledge base that can be investigated and searched and utilized and better understood and valued and weighted and things like this. And so I'm excited to see the improvement of citations from scite, but also from others. And I am also a big fan of Garfield. I only got to speak with him on the phone.
JOSH NICHOLSON: But when I close my talks where I have slides, I generally show this PDF from 1964 where Eugene Garfield talked about adding more context to citations, and he called them citation markers. And I think it's been a challenge to do that because the scale of scholarly publishing is quite large. But I think with tools that have become available, we're now at a stage where small startups like ours can do really kind of innovative things.
JOSH NICHOLSON: And I think this will spread out into other areas, other types of data-- so not just looking only at scholarly publications, but alt metrics, I think there's room for improvement and things like this.
SAI KONDA: Thank you, Josh. Marie?
MARIE MCVEIGH: I think-- obviously we're a couple of Garfield fans here. But I think one of the things that I want to raise is the fact that citation itself was originally, as Rebecca said, it's individual and narrative of its own nature. It is-- we talk often in the selections group about the global scholarly dialogue. This is a conversation. So originally, citation metrics and the idea of citation indexing was based on the fact that Garfield saw value in this conversation that we could now explore at scale and also break some of the boundaries of time.
MARIE MCVEIGH: When you're reading a paper you go back and look-- you go back and look at the things that it referenced, but when you start to index and aggregate, then you can actually say this paper became these other things. And I think one of the insights that you're beginning to leverage in a new way is to say that information itself is plastic in that it changes as you use it.
MARIE MCVEIGH: The notion of citation indexing was based on the fact that if a paper is written, it can only contextualize itself based on what it knows up to that day. As the field evolves, as that paper is used and cited and discussed, then that paper acquires additional meanings which the index are at the time of-- or the author at the time of writing cannot predict and understand.
MARIE MCVEIGH: So that indexing actually is a way to draw information forward and continue to modify its meaning, show how it has evolved. So that was the underpinning of citation metric. The problem is, the awesome power of having 150 million of these highly meaningful interactions became a kind of statistical juggernaut and people stopped looking at that conversation, although the impetus was that conversation.
SAI KONDA: I think this is great, because I see context and narrative being a common theme in what all three of you have just spoken about. So why don't we just unpack that a little bit more. Just thinking about context, how do you think-- and also in theme of the New Directions webinar as well, how can we make best use of these metrics, number one, given the fact that, as Josh mentioned, we have a billion of them at this point? What is the best use for them?
SAI KONDA: And also, I would also like each of you to give your thoughts on what is the narrative here and how it can be applicable in that situation. Let's start with Rebecca, maybe.
REBECCA KENNISON: So I'd like to-- before I answer that question-- or kind of along with answering that question, I'd like to put in perhaps a bit of cautionary note. I agree with everything that Marie just said and that Josh said, but I am concerned about the people who aren't part of this conversation. So even if you have written a piece and it is now published and it's out there and we know that, again, thinking about humanity's work, some of this never gets cited at all.
REBECCA KENNISON: Does that mean that you're not part of the conversation? Does that mean that-- it certainly in a metrics way is a downside. Because what you need to do if you're an academic is report on not only what you published, but also how that's impacted your field, and the impact almost solely relies on who might be citing you.
REBECCA KENNISON: Except for when you have external review letters, like when you're going up for tenure or when you're going up for promotion, you get review letters where people talk about your work, and that might be the only time that some people talk about your work in that sense. And so while, again, I think the whole conversation is important, but if you're not being part of that conversation, I think that that can be a concern.
REBECCA KENNISON: I'm also concerned a bit about where work might be presented and how it might be presented in a way that is not part of the formal publication record. And I know that there are ways that people are looking at and capturing-- it's that kind of information. If you write a really well-regarded blog post-- or I don't know, a Twitter thread that is really used and could be a citeable conversation, how does that go in?
REBECCA KENNISON: And thinking more broadly about the metrics and the conversation and the narrative and everything. It's the full package of that conversation, it is not just within citations and not just within-- that scholarly conversation is only one part of the conversation. And all metrics, of course, have tried to try to look into uncovering some of that other conversations.
REBECCA KENNISON: But everything that I've just raised is also part of the metrics concern. Like some people really are very good at being out there and in social media and some people really aren't. Does that mean that you are not part of that conversation? And I'm not sure that I actually-- I kind of danced around to the question there, Sai, a bit, but I think that all of those things are really kind of important and I would like to hear my co-panelists unpack that a little bit more.
SAI KONDA: No, thank you, Rebecca. I think actually unpacked one thing that I was going to follow up in a few minutes later, is you mentioned impact. And I'm going to put that in quotes, and what we do really mean by impact. I think that's worth having a discussion right now since it's come up. Josh, Marie, if you have any thoughts on that.
JOSH NICHOLSON: Yeah. I think impact can mean so many things to many different people. I try not to use it. I think generally, though, to kind of come back to your question, most measures and metrics that we're looking at is looking at how many as opposed to how, and I think starting to see versus how many, whether that citations or tweets or mentions in policy documents, I think will start to provide more nuance and context and understanding.
JOSH NICHOLSON: And I think will start to allow us to use these metrics in new ways. So one thing that we see at scite is a lot of students using scite daily for their essay writing because they can see what have experts, what have PhDs or grad students written about this topic that I need to write about? And I would say they don't traditionally use bibliometric tools.
JOSH NICHOLSON: They're probably not going to Scopus or Web of Science to help them with an essay unless they're doing bibliometrics work. And so I think this data, whether that citations from scite or tweets from Twitter, can be used in new ways as we start to surface it in new ways, and I think that's particularly exciting because really, it is like people's writing, this conversation shouldn't just be boiled down to, here's a linkage, here's some impact, but guess what impact it is?
JOSH NICHOLSON: Or maybe don't even guess because it's too much work to kind of investigate that manually. Yeah.
SAI KONDA: Thank you, Josh. Marie?
MARIE MCVEIGH: I would love to see impact take the lesson of citation metrics, which is harvesting these small point-to-point interactions and saying, what can we do at scale to understand the landscape? And also, never erase that original aspect of that dialogue-- of that narrative. I'd say I love to-- I'm looking over here because I have the other screen where I pulled up the Merriam-Webster definition of impact and just lost the Zoom session.
MARIE MCVEIGH: And impinging or striking especially of one body against another, or a forceful contact or onset. Also the impetus communicated in and as if such a-- as if in such a context. I love that idea of this influence, this moment when you have changed your environment by the force of your idea, of your citation, of your presentation, of your panel. When you make people think in a different way, that's impact.
MARIE MCVEIGH: Now I would like to see that we could take this more generalized notion and say, how do we also bring this to a fact base that we can share? One metric being not perfect for everything doesn't mean no metric is perfect for anything. I wrote an essay a couple of years ago where I talk about the metaphor of impact or the metaphor and metrics where I talk about, let's look at thinking about metrics generally as a mosaic.
MARIE MCVEIGH: How do you take these individual things which don't capture the whole picture and assemble them so that you can look at something incredibly complicated, like academic production or academic performance? You don't use one tile for a mosaic, you use many, because the individual tile does not convey the meaning, but the assembly of them in a meaningful way is what contains that meaning.
MARIE MCVEIGH: So any metric has a context in which that metric itself can reflect some aspect of impact. But the picture as a whole is a much more nuanced thing.
SAI KONDA: I love your analogy about the mosaic and the tile view of looking at things. So I actually want to bring it back to where the impact-- where I see impact being used is the journal impact factor, obviously. And I think-- I want to hear all of your thoughts on this because as I see it, at the end of the day, it's a mathematical construct, it's a simple ratio, but then you translate that into-- and give it a term as a journal impact factor, it means a whole lot of things for different people.
SAI KONDA: So when especially when you talk about this mosaic picture and the journal impact factor being just one tile in that mosaic, how do we go about thinking about it, especially and also using the term context that all of you have mentioned before, what's the best way of really digesting this information whenever these numbers come out on an annual basis? And what it means, for example for humanities versus other areas?
SAI KONDA: What does it really mean for different areas?
REBECCA KENNISON: So the impact part-- I just want to-- that's a very nice lead-in, is if you know exactly what I'm going to say, which you don't, necessarily, which is that the time frame question, I think, is a really important one, and this is why I like the mosaic metaphor as well. Because impact factors are just like a three-year window, and in the humanities, of course that can be an extremely short window.
REBECCA KENNISON: It takes a very long time for ideas to percolate, and sometimes you don't see impact. But I'm also just thinking, because the Nobel Prizes are coming out, I'm reminded every year when the Nobel Prizes come out that those are like Lifetime Achievement Awards, if I can put it that way. They're for work that happened decades ago or decade-- or a decade ago.
REBECCA KENNISON: So even within, when you think about the impact of those particular people's work, it's only now that you're starting to really see that impact happening. I agree with Josh that I'm not sure impact is a word that kind of avoid a bit because I'm not sure-- I like the-- the dictionary definition, too. But impact on whom, by what.
REBECCA KENNISON: Because at the end of the day, we're talking about people doing-- having conversations and what is changing. And sometimes you can't capture that like I read a piece, and let's go back to citation practice. I read something but I don't publish a lot, but a lot of things influenced me. So that's an impact on me that's not being captured out there in the general conversation.
REBECCA KENNISON: But then there are also-- I'm thinking about Judith Butler's work, for example. Gender Trouble, which has been very transformative in the way that anybody thinks about gender construction. But that took 10 years before anybody really started to pay attention to that work. And now it's transformative, but along with everybody else. I mean-- and then the citation to Judith Butler happens, but not to all the colleagues who were also working within that context.
REBECCA KENNISON: So it's kind of a shorthand. So does that mean that Judith is more impactful than somebody else? Again, I'm troubling the waters here by having these-- by thinking about it this way, but again, who gets to be the marker for this impact I think is really important.
SAI KONDA: Thank you, Rebecca. Josh?
JOSH NICHOLSON: Yeah. So the impact factor and citation counts in general I think it's why I started scite. And so I think there is-- about six years ago, studies coming from large pharmaceutical companies and nonprofit initiatives looking at reproducibility in science. And I knew within my narrow field of chromosome segregation and aneuploidy which studies had been tested and either shown to be right or wrong.
JOSH NICHOLSON: But if someone else came into my field, you wouldn't know this information, and you'd see this highly cited paper with 300-plus citations. And so really, what we wanted to do initially and I think why we've started to do it is to say, hey, not all impact is kind of the same, and not all things that we're tracking, whether that's impact factor or total citation count or h-index, we're only looking at the quantity, not necessarily the quality.
JOSH NICHOLSON: And so wouldn't it be great if we could start to show more nuance, more understanding, more interpretation from other experts? And I really like, that's why it's this kind of continuation and it adds so much more even after publication. And so I think, as probably my-- I don't want to say hate, but like disliking of just how simple we use citations that led us to actually trying to improve upon that.
JOSH NICHOLSON: And it's kind of funny, because in that paper that I initially wrote saying, here's a better way that we could use citations, I suggest all these different groups should do it, including ISI and Thomson Reuters and ultimately that's not how things generally get done. And so we kind of stumbled into this. And I think what we've done now with citations I think ultimately will be done by all citation tools. And I see Clarivate already starting to show some citation context, and I think it's exciting to see that, because there's so much value in there that is lost on just a simple number.
JOSH NICHOLSON: And I think-- yeah. And that's true of impact factor we can hopefully--
MARIE MCVEIGH: I'm going to say, if you're not careful, you're going to start recommending that people actually read the paper.
JOSH NICHOLSON: Yeah.
SAI KONDA: Marie, any comments from you?
MARIE MCVEIGH: Only that there is-- we talk about the journal impact factor as though that were the only data point that we publish or the only metric that we publish. I did a calculation a long time ago-- and I will have to update it, but it gets a little intimidating, and that is at the time we published about 12,000 journal impact factors in that year. And I went and calculated how many data points were in the JCR that year, and it was over 8 million.
MARIE MCVEIGH: There is an incredible wealth and texture of data that we provide, and it's not two years long, it's 10 years long with indicators to any out-years further than that. We indicate that there is a greater depth of historical use of a journal through a metric that extends-- we had a cited half-life, the average age of materials in a journal that were cited in-- I think was the year 2017 data, that cited half-life of that journal was over 80 years, which means that 50% of the citations to that journal in one year were to materials that were more than 80 years old.
MARIE MCVEIGH: So it's not that we are not aware of that and that's not that we're not serving that up at a level of depth and texture. If all you're looking at is the one tile, you're not seeing even the mosaic that we're presenting in the one product where we contextualize that metric. It's not like, here's your impact factor, goodbye, go away.
MARIE MCVEIGH: It's like, no, here's where it came from, here's where it goes, here's what built it, here's the network of references. Rebecca with the ANH Journals in the JCR this year, there's a full citation network of cited and citing materials that also represents nonindexed materials. So everything is in there to create that context around, where is this journal in a year participating in the literature?
MARIE MCVEIGH: What does it reference, what reference is it? How do you start to get at where this individual item is placed in this broader context? Individual item being the journal and all of its content.
SAI KONDA: Thank you, Marie. All relevant points. I think there's a question that came in from the audience. And I think this is kind of related to what you just mentioned, and I wanted to put it out there in case others have thoughts on it. And the question is, as mentioned, impact factor is only one type of metric as we've been discussing, but many researchers and institutions laud it as the most significant and most important.
SAI KONDA: How can we better inform researchers to understand it is only one tile in the mosaic? So he said-- it's talking about the mechanism as to how we can educate people at this point.
MARIE MCVEIGH: Oh, golly. 25 years now I've been educating people. I still have people who don't understand [INAUDIBLE].. You said it was a ratio, bless your little heart, because it's been-- yeah, it's been called an average. So I think one of the terrible warnings to metrics that are coming down the pike is that if it's simple, it will be misunderstood and misused. If it's complicated, it won't be understood at all and it won't be used.
MARIE MCVEIGH: So how do you start to create this dialogue of the metric and its meaning? And I think that's really why I joined Clarivate, is to bring that in there and say, this-- OK, you have a number, but here it is in a five-year graph in the context of how this puts this journal in a category perspective. And here's all the articles and all the citations that created that number.
MARIE MCVEIGH: Oh, by the way, it's only 10% of the total citation activity. I look at the JCR as a way to look at a much more complicated picture of the value of a journal and that journal as a part of the global scholarly network and begin there and then use that. Go elsewhere from this. This is not one thing or one metric, this is one place to go for information, and then to take that much more nuanced view about, what are you looking to really value?
MARIE MCVEIGH: Do you need open access? Do you need timely turnaround? What is the real content of this journal and how does it perform? And how much is that impact factor based on one article with a large number of citations or a broad and deep-- basically a deep bench so that you can-- your journal is actually a continuously evolving set of valuable content?
MARIE MCVEIGH: So yeah, just-- this is a terrible thing, but look at the metric in the context in which we provide it. It's not by itself. We never do that.
REBECCA KENNISON: If I could jump in and say, I think that back behind that question is, why do people go to proxy-- why people go to proxy measures? And that is something that we all agree that it shouldn't. But it's not that people don't know that there are problems or so on. It's just that there's a vast amount that's being published and people are just trying to get a handle on, again, what is the-- yeah.
REBECCA KENNISON: What does any of this mean in terms of faculty productivity in particular? And so it becomes a question about who's going to get promoted and raises and so on, which is a very different kind of question than, what is the real impact? What is the real conversation that's happening and so on? And that goes back to the need for more narrative, more context, and reporting that out.
REBECCA KENNISON: But again, that then becomes a responsibility of the researcher. And then I want to stop and say, when we say the researcher, that's also what is being prized. It is the research side. But that's only one part of the enterprise, which has this three-legged stool of research and teaching and the big bucket of service where peer review and everything falls into that.
REBECCA KENNISON: And so one of the things that my colleague Chris Long, who's the Dean of the College of Arts and Letters at Michigan State, is doing is actually thinking differently about how he asks his faculty to describe the work that they're doing. So the work that they're doing, don't describe it as, what did you do in terms of research? What did you do in terms of teaching? What did you do in terms of service?
REBECCA KENNISON: But think about something like, how did you share knowledge this year? Now tell me how you shared knowledge. And that could be in your research and your publication. That could be in your teaching. That could be in your mentoring. That could be in the community conversations that you've had in your community engaged work. That could be anything.
REBECCA KENNISON: And it really has been very transformative for his faculty to think about things in that lens rather than how many pieces did you publish? How many classes did you teach? And how many articles to do review and so on. So it's a different way of looking at it, but I think that also helps-- if administrators would start to think about it that way, then those numbers are completely different.
REBECCA KENNISON: I mean, who cares about the journal impact factor within that context? It's not necessarily about sharing it.
SAI KONDA: Oh, that's a great point. Great initiative. And it almost takes me to the term that Josh mentioned in terms of conversations that's-- it's, how many conversations is your work kind of encouraging in the community? Josh, any closing comments on this particular topic?
JOSH NICHOLSON: Well, my neck was probably hurts from agreeing so much with what Marie said. Because I think-- the impact factor sticks around because it's simplistic and easy to understand, and we do need some of that. We're all overwhelmed with tons of things to do more than ever. And I think there have been probably way more sophisticated, much better metrics than the h-index, the impact factor. But oftentimes they're lost right.
JOSH NICHOLSON: And administrators and the people that are using these things, you have to think about, OK, who's the ones misusing it? Why are they misusing it? Can we make things more easy for people? There's this ideal world that we all want, but there's also reality, and you have to kind of go towards that. And I think acknowledge that we do need some proxies, we do need some heuristics to help these people, because it's impossible to sit around and read every single paper, watch every single talk, be inside someone's head.
JOSH NICHOLSON: And so looking at these things from kind of that viewpoint I think will help us design and kind of come up with better metrics and proxies, but I think there needs to be some realism in that. It's not enough to just say, we shouldn't have any of this, we should do all that because that's-- it just won't happen. And we use metrics outside of scholarly publishing for many things that we do and we appreciate that. And of course, there will be abuse of things.
JOSH NICHOLSON: We're all humans. There's abuse of everything. But I think-- I do think there's a lot of value in metrics to kind of round that back out. And so I really like this tension between simplistic and being adopted and way too sophisticated and no one ever uses. There needs to be somewhere in the middle, I think.
MARIE MCVEIGH: What about-- I don't know. One way that-- for good or ill-- and I'm really shooting from the hip here-- all metrics are in some degree a proxy and a part, not any individual. So I think we've got that kind of nailed amongst us. But what about the idea of metric as a possible filtration mechanism? So if I am interested in-- I'm looking for a journal to publish and I'm looking for journals in my field, I want to find the OA journals.
MARIE MCVEIGH: How do I find them or those journals that have a significant amount of this content? Well, there's a metric for that. And it's a way to lead in. It's not a way to nail down or close that door, it's a way to find an initial thing. And then you can dig deep into that context or into that situation. So how do you find a paper?
MARIE MCVEIGH: You can find it by a Google search, you can find it on Google Scholar, you can-- I don't know what their ranking is, but you can find it. You need to dig in to individually value that paper. How do you know amongst 3 million results you get when you search actin binding protein, how do I find the paper that's going to be relevant to me in this moment? And there's a lot of different mechanisms around that and a lot of different ways to start thinking in that direction.
MARIE MCVEIGH: So can we think of metrics as a filtration or as a way to identify outliers? This is something novel, this is something new. A metric of novelty is something everybody's been lusting after and unable to do because it requires time, and then it's not novel. So do we have-- do we have a way to move in that direction?
MARIE MCVEIGH: Josh, I'm really looking at you.
JOSH NICHOLSON: Sorry, I got distracted by the chat, and I was starting to read all those things. And so now I-- so I think-- I agree with kind of what you said, and I think how we get to where we're going and what we measure will be determined by kind of like, A, what is possible; and then B, what can be useful and also understandable? And so yeah.
JOSH NICHOLSON: I don't have too much to elaborate there because I got distracted, apologies.
MARIE MCVEIGH: I like that, what is possible and what is useful.
JOSH NICHOLSON: Yeah.
MARIE MCVEIGH: Because what is possible and what is useful is often a far cry from what is ideal and what we really want, but we should always be aiming towards that? What is ideal and what do we really want? So let's begin here and move in that direction. Yeah, I haven't looked at the chat and now you're making-- now you're making me scared.
JOSH NICHOLSON: Yeah.
SAI KONDA: Rebecca, you wanted to add something? Otherwise I was going to bring up the things from the chat.
REBECCA KENNISON: No. I just-- I was interested just to comment on the novelty question, which I think novelty and originality or interesting concepts when what we're talking about is that all of us are in a conversation with each other. And so I would just like to say I don't know how novel is any given idea when we're all building on the work of everybody else. But anyway, I'm just tossing that out there, as I think-- I'm not sure how one would assess the novelness or the originality of a particular work when the whole point is to build on everybody else's work.
REBECCA KENNISON: There. Sai, what do you want-- what do you want--
SAI KONDA: I think we can have a separate panel on the topic, I think. So those are very good questions that came in from the audience that I wanted to forward to you. So it's about, what about data citations? I think we spoke a lot about research citations. And are data citation tools like data citation index being used? And if so, how and by whom? Anyone would like to go first?
MARIE MCVEIGH: Data citation index, yes. We have been building that out for, what, 10, 12 years now? They started before I left, they were still at it. So we tried at that point to both identify and promulgate the idea that this is an important part of how this research is done. This is a necessary point of information. How do you recognize that? By creating a pointer to that. So the problem is in part the availability of an identifier.
MARIE MCVEIGH: And I will-- it might be the last Garfield quote, but it might not. In his 1955 essay, he actually says, oh, gee, wouldn't it be great if all the publishers would include this like code in the reference list that would indicate what this article is? And here's an idealized data extraction that it could be. And then everyone could do this group assembly of this stuff. I'm like, ooh.
MARIE MCVEIGH: DOI in 1955. So we needed a mechanism for data citation. We needed a way to identify that unique thing which is the entity cited in a data citation. So there's this kind of Necker cube of, well, you create an identifier for me first. Well then, I'll be able to cite it. Well then, you cite it so that I can create an identifier. But both of those things are progressing in parallel.
MARIE MCVEIGH: So I think we're moving in that direction. And again, once we get this at scale and a more generally adopted practice, we're going to be much further along. But the mechanism is starting to come into place. So there's an awful lot already there, and I think that we've laid the groundwork-- we as a community have laid that groundwork to start moving that forward more aggressively and more-- to create more use and more usefulness.
MARIE MCVEIGH:
JOSH NICHOLSON: Yeah. I mean, I guess I can add to this. So we-- this is a limitation of scite. So there are DOIs for many data sets from data side. There's different DOI registries, though, and there's the practical limitation of where can we get all this metadata from all these different areas? And so we don't do too much with data citations at least at scite.
JOSH NICHOLSON: But one thing that I can also mention, which I think is interesting, is we have this citation statement search. So you can search these fragments of text, extract it from the full text, and oftentimes this is used to say, I want to put in this URL for this data set to see how has this been used, because it's not in the reference list. And so I think there's still a lot of learning in the community to how-- where do I put my data, for one?
JOSH NICHOLSON: How do I make it referenceable? Like how do I then reference it myself in publications? And I think we're still pretty early there, at least from what my personal experience handling data and publishing data, which really kind of didn't happen much, to actually just using this search and seeing how other scientists reference it. They link to it, there's repositories, but it's still kind of treated differently.
JOSH NICHOLSON: And I think that will start to change. I don't know how quickly, though, and I think that comes to like, what's in it for me? There's a lot to do as a researcher or author now. This could be one more thing to maintain, one more thing to look at. And so you have to kind of align that as well and think about, OK, why is this person going to share their data?
JOSH NICHOLSON: Just for the good of the world? Is it going to benefit them? Do data citations count? Who's counting them? All these things. And so I think the groundwork is there, but I think we're still kind of in the infancy of looking at data citations, at least from my perspective.
MARIE MCVEIGH: It took 45 years to get a DOI. We had one internally. We're using internal codes, and it's funny, because that it had to be there, so Dr. G invented it. It just didn't get adopted.
REBECCA KENNISON: So when I was at PLOS a long time ago, when PLOS first started, we started by granularly identifying the idealized-- the graphics and then everything. Which was-- Crossref was like, we don't know how to handle this, but [INAUDIBLE], right? So to what Marie was saying, yeah, it's been a long time that we've been having like how do you persistently identify something that is data or-- But the problem has been, as you said, Josh, where is this data located?
REBECCA KENNISON: And also, again, going back to the context question, there are a lot of concerns particularly-- in this case, in social science about misinterpretation of the data, if it's just the data set without the context. And so in order to get people to give us data which we wanted in the repository at Columbia where I was working, we really did want data, I would tell them, well, give us your data and then give us everything-- all the stories that you've told about it in whatever format, wherever.
REBECCA KENNISON: And we'll put them all together and we'll package them together so that you can have the context. And that really did help, but I do think that there are some concerns just about what-- there's the persistence questions, like where is it and how can you find it? But then there's also the context question. And again, I think those things still need to be solved. In the same way as we've already been discussing about the whole record.
REBECCA KENNISON: There needs to be context, it needs to be findable, it needs to be persistent.
JOSH NICHOLSON: Persistent, I think, is an important one. Because-- so GitHub, where a lot of data lives, just introduced this ability that makes it easier to cite a GitHub repository. But GitHub is not, [INAUDIBLE]. I don't know the persistence of this. Maybe it is, maybe it's fine, but I think it doesn't come with a lot of these quality assurances that maybe less-used repos do. And so already there's some friction of like, OK, if I need to cite my data, can it be where I do all my work or I have to move it over to this other system that I don't use?
JOSH NICHOLSON: And so I think those things seem very small, but they matter a lot. And I think we get a lot of questions about scite and our machine learning and all this sophisticated stuff, but I think what is equally important-- and this ties into metrics in general, is how easy is it to use and understand? And I think that will be necessary for depositing data and citing data.
JOSH NICHOLSON: Because right now it's confusing. There's like this whole guideline of how you cite software as well, there's principles for it, even. And that friction will lead to like just the lack of adoption, I think, or slow the adoption.
SAI KONDA: I know we have seven minutes left on the clock. And so I want to maybe get one question in. Since DOI was mentioned multiple times, I thought this is a relevant one. So this is going back, there's a question from the audience on Rebecca's point on data mining, impactful citations. Citation metrics differ substantially across Google Scholar, Microsoft Academic, and Web of Science. Do we measure citations only by citing work as a DOI?
SAI KONDA: Would preprints be counted as valid citations? And what about grey literature?
JOSH NICHOLSON: So I-- yeah. So at scite, we use only DOI, and that's just a practical limitation. Like it's a great resource. There's 120 million records with metadata tied to it, but without DOI, scite probably would not exist. It would just be way too much work. And so there are other sources that are extremely valuable. Grey literature, I come from another company that I started called The Winnower, which was exclusively giving DOIs to gray literature.
JOSH NICHOLSON: So to blog posts, to Reddit science AMAs, all these things that didn't count because they weren't being cited. We tried to say, hey, this should count, and here's a way to make it easier to count it. And so I think there's practical limitations of harvesting all these things. It's not technically easy to do. There's not a lot of citation indexes out there because it's not something you just spin up and magically are grabbing information from all around the world.
JOSH NICHOLSON: It's Google Scholar, it's places that have been doing this for decades. And so I think those things do matter a lot. Preprints, I would also note, a lot of them have DOIs, a lot don't. And so I think this is something that is, when you're considering where you publish, you think about, OK, well people read this, and I think the same is true for kind of the data you use and the venues you put.
JOSH NICHOLSON: And personally, I probably wouldn't publish publishing something that didn't offer DOIs because I would know it would be harder to cite or could not be cited in some cases. But I do think there are new tools and efforts, and all kind of tools will start to explore this. Overton is one, that's looking at citations and policy documents.
JOSH NICHOLSON: And so starting to look at areas where we haven't-- it takes some effort, but it's definitely valuable and worth doing.
SAI KONDA: Thank you--
MARIE MCVEIGH: I'm running a reference search to see how many times The Winnower was cited in Web of Science. I've got two pages of versions of references, some with 15 citations.
JOSH NICHOLSON: Yeah.
MARIE MCVEIGH: That's the other bit of citation indexing. Is it's not about DOIs, which make it easy-- anybody can run a citation index on DOIs, but we were running a citation index for 45 years before there were DOIs. And it was about capturing the thing that is referenced in a metadata sense so that it can be unified, not that every occurrence-- no matter how unique the way that thing is referenced, oftentimes we are able to glom those together and say, this may be cited with a URL and then it cited without a URL, but we know that those are the same thing.
MARIE MCVEIGH: So we're probably looking at 200 or 300 citations in Web of Science that are trackable to-- and there were some with URLs and with-- I don't know if they have-- yep, some with DOIs, they're right there. So that idea of there is that visibility on that network and it has-- it's more than just a DOI, because not everything that's worth referencing and not everything that enters the scholarly literature as an item of discussion-- I always joke about Beyonce's Lemonade album as a feminist work, she doesn't have a DOI.
MARIE MCVEIGH: But she has a record in Web of Science as a cited work, because that is part of how the literature is evolving and using that work. Now I have to finish the search. I have to know how many times.
REBECCA KENNISON: Anything can have anything can have a DOI, but again, not everything gets indexed properly. And not to go down the metadata rabbit hole, but that's how-- when I was running the repository at Columbia-- not I-- my team was running the repository at Columbia, we spent a lot of time working with Google Scholar to optimize the metadata so that they would pick up everything that we have [INAUDIBLE]. And then we went and asked people for their blog posts and so on so that we could hold them in the repository so that they at least would have a [? DMI ?] and be citeable there.
REBECCA KENNISON: But working with everybody on what kind of metadata structures they need and so on I think is really, to Marie's point, the reason you can find Beyonce's album is because there is metadata about Beyonce's album. And the way that people cite it, there is a regularized kind of way that people can cite that. And so I don't want that to be the final word on anything, but to say thank you all the people who are working on metadata and metadata standards and enriching that metadata, we couldn't do it without you.
SAI KONDA: Thank you. And I think we will call this to a close with that note. I would like to-- great thanks to all three of you, Josh, Rebecca, and Marie, for very insightful comments. And thanks to the audience for sending all your wonderful questions. I personally learned a lot. I have a long list-- mental list of things to think about, one being to read more on Eugene Garfield's words.
SAI KONDA: Thank you, Marie, for mentioning. But I just wanted to say thanks to all of you. And just as a note to the audience, we have a 15-minute break following this session, after which there will be session on funds, funders, and funding. So please stay on, catch a break, and then jump back online. But thank you once again for your time, and hope everyone had a great fun listening to this panel.
SAI KONDA: And have a good rest of the day. Mary Beth, any closing comments?
MARY BETH BARILLA: No. Thanks, everyone, and we'll see you back in 15 minutes.
SAI KONDA: Thank you so much, everyone.
MARIE MCVEIGH: Thanks, everybody.
Segment:0 .
MARY BETH BARILLA: Hi, everyone. Welcome back to day 2 of the SSP New Directions seminar. If you were not able to join us yesterday, my name is Mary Beth Barilla and I'm the program director at SSP. I want to thank and acknowledge our sponsors again today. Access Innovations, Atypon, Cadmore Media, eJournal Press, Impact Journals, Jack Farrell & Associates, Silverchair, Straive, and Taylor & Francis, F1000.
MARY BETH BARILLA: Thanks so much for your support. We've got some housekeeping issues for you. At any time during today's sessions, if you need support, life support is available. Just go to that Support tab there at the upper right in Pathable to find a link and it'll connect you with a person. If all else fails, customer service's email is info@sspnet.org. We will enable closed captions for our sessions today, and you should be able to access that CC button on your Zoom toolbar.
MARY BETH BARILLA: We are also recording today. And if you miss part of a session or want to come back and listen to a particularly interesting session, you'll be able to do that. We'll post those at the end of the seminar. And we do want to hear from you again today. So if you Tweet, our hashtag is #sspND2021. There's a Q&A box to direct questions directly to the panelists.
MARY BETH BARILLA: There's also a chat feature. Be sure in the Zoom chat to select everyone if you want to talk to all of your fellow attendees, or just select hosts and panelists if you prefer. But just be mindful, you have two choices there. And now I'd like to turn things over to Sophie Reisz, Working Group Lead for our New Directions Working Group.
SOPHIE REISZ: Thank you very much, Mary Beth. Hi, everyone. I'm Sophie Reisz, Vice President and Executive Editor at Mary Ann Liebert, Inc Publishers. It is my great pleasure to introduce our first session of the day, What's the Issue? Impact and Metrics, moderated by Dr. Sai Konda. Sai is the Senior Managing Editor in the Global Journal's Development Publications Division at the American Chemical Society, and is also an SSP board member.
SOPHIE REISZ: Sai, over to you.
SAI KONDA: Thank you, Sophie. Warm welcome to everyone. Welcome to day two of the New Directions seminar. Just before we get started with introductions from our panel of speakers, I wondered also give a shout out to Alexa Colella from University of Illinois Press for getting the session together and putting the panel together. So thank you, Alexia, in addition to all the members working behind the scenes. So just to get started off with the session, I wanted to get some introductions from our panel of speakers.
SAI KONDA: If you can briefly introduce yourselves and talk about your roles in your respective organizations. Maybe we can start off with Josh first.
JOSH NICHOLSON: Great. Thanks for having me, and thanks, everyone, for tuning in. My name is Josh Nicholson and I'm the co-founder and CEO of scite. I have a background in cell biology. So I finished a PhD about six years ago. I have always been interested in kind of how science is done, which is, I think, led to where I am today. And I feel like I have this kind of love-hate relationship with metrics and citations where I kind of hated how they were and how they're used in some cases so much that I've tried to do something about them with scite.
JOSH NICHOLSON: And I think now, used in the right way, they can be incredibly powerful, and so I'm excited to talk about our use case at scite and what we're doing. Yeah.
SAI KONDA: Thank you, Josh. Rebecca, next.
REBECCA KENNISON: Hi, everyone. I'm Rebecca Kennison. I'm the Executive Director of K|N Consultants, which is a non-profit agency that works with higher education institutions and organizations, including academic publishers and libraries for all the librarians, to enact transformational change. And then my background is both in publishing. I was an academic publishing for a very long time, including open access publishing.
REBECCA KENNISON: So yesterday's sessions were really, really interesting in that regard. And also, I worked in an academic library for eight years. So I think I understand both those sides very well. There are a couple of projects that are going to help to inform what I'm going to-- some of my comments, whatever my comments might be later. Both projects were generously funded by the Andrew W. Mellon Foundation.
REBECCA KENNISON: But the first one was publication ethics, which I worked on with several philosophers and with the executive director of the American Philosophical Association. And in particular, that project, while it looked at publication ethics of all kinds, it was very interested in bias and citation practice. And so in the humanities, and for people who know philosophy, philosophy is the straightest, whitest, oldest of the humanities.
REBECCA KENNISON: And so it was a really interesting project. And it did really transform my thinking in how I look at citations and think about citations. And so there's that one. The second project that I'm currently involved in is the Humane Metrics in Humanities and Social Science Initiative or Humetrics where I'm a co-PI. And among other outcomes, we're looking to shift thinking in the academy so that we focus on measuring what we value rather than valuing only what we can measure.
REBECCA KENNISON: And so there will be a lot of that aspect that I also bring into my comments today. Thanks.
SAI KONDA: Thank you, Rebecca. And last but not the least, Marie.
MARIE MCVEIGH: I was hoping you would say that so I wouldn't have to or so that I wouldn't feel least. I really appreciate being part of a panel where in some sense I do feel least, because I really respect these two individuals and have followed their work across years. Background, I joined Clarivate in 2017 as part of the development team for the Journal Citation Reports.
MARIE MCVEIGH: I wanted to bring a new sense of that dynamic relationship between article performance and journal performance. And to open up a much greater transparency on the components of the journal impact factor, because that had been such a matter of discussion and debate and dismay. In 2019, I moved to the editorial team to continue what has been a career-long interest in bibliographic and bibliometric research.
MARIE MCVEIGH: And to work in the integrity and maintenance of the scholarly record that is reflected in the Web of Science Journal resources. Before Clarivate, I worked in a bunch of roles in primary publishing, in A&I analytics, and author services, in editorial support systems, but I really started this whole adventure at the Institute for Scientific Information as the Selections Editor for the Life Sciences Content, coming out of a cell biology degree, I will say.
MARIE MCVEIGH: I was there for the first launch of the Web of Science. I've also, for about 10 years, I led the production and product development to the Journal Citation Reports prior to its new installation. And I've also-- I started up there doing surveillance and review of citation metrics with the manipulation of metrics, with the use of detection on the misplaced priorities that researchers bring to try to juke that system.
MARIE MCVEIGH: At ISI, I had the privilege to study under Dr. Garfield. And I continue some research into bibliometrics and ethics and publishing and citation metrics. And the real value of citation indexing, I'll say, I let Garfield's original intentions for citation indexing be kind of a lodestar for me. I'm still an idealist all these years later. And what is really important underneath all of that is that this is built on a fundamental aspect of the scholarly dialogue.
MARIE MCVEIGH: And I think we all agree that that itself is a point of value.
SAI KONDA: Thank you so much, Marie. It's great. I think it's already evident given the diverse backgrounds and experience of our fantastic panel of speakers. So before I jump into my open-ended questions to the panelists, just to remind the audience that please feel free to use the Q&A box in Zoom to forward your questions. We will try to take them during the discussion as humanly possible. If not, we'll try to have a Q&A session at the very end so that we can have a discussion on different topics.
SAI KONDA: So let me leverage all your diverse experiences and start off by asking you what your perspectives on where metrics stand and why do you think metrics are important? Let's start off with Marie.
MARIE MCVEIGH: OK, first and least. I'd say that metrics in their best instance provide a verifiable checkpoint. And I'm talking about metrics generally. We'll touch citations throughout, I'm sure. Verifiable and transparent doesn't make a metric perfect, and it doesn't make it fit for every purpose, but it can create a shared fact base. Which is why I like metrics, it's a place to begin to put things on the table and say, this is what we are looking at and this is what we can begin discussion with.
MARIE MCVEIGH: Like any data, metrics are not a substitute for judgment, they're a tool to guide and share judgment and to share that sense of fact base. They are support and an opportunity to generalize the principles that underlie a decision. And I want to put like right upfront why do I think metrics are important. I want to put upfront at the opening of the Leiden Manifesto, which was-- I visit again quite regularly, research evaluations that were once bespoke and performed by peers are now routine and reliant on metrics.
MARIE MCVEIGH: The problem is that evaluation is now led by the data rather than by judgment. Metrics have proliferated. Usually well-intentioned, not always well-informed, often ill-applied. We risk damaging the system with the very tools designed to improve it.
SAI KONDA: Thank you, Marie. Rebecca, next.
REBECCA KENNISON: Just to piggyback on what-- very provocative there, Marie, in terms of-- I think what people understand is that metrics-- like everything, including impact factors and so on, is designed to solve for problems that would help this to be a much more equal and equitable environment in which to operate.
REBECCA KENNISON: And what has happened over time, as the Leiden Manifesto points out, is that we've become data-driven instead of data-informed if I could put it that way. And we don't necessarily understand the context within which we're seeing whatever that number might be. And whether that's an h-index or an impact factor or just counting up citations and so on, I mean, as I mentioned, one of the things that I'm really interested in is citation practice.
REBECCA KENNISON: And in the humanities, this is a particularly important thing, because that's all you got. Basically in the humanities is you have a conversation between this piece and now what I think and then people-- now then talk about that piece and so on. This is the way that humanities in particular-- I mean, all of the academic conversations go this way. But this is the sole way that the humanities do their work.
REBECCA KENNISON: But who gets cited, who doesn't get cited? What are the practices that publishers have imposed basically on this? Like limiting citations, which we don't need to do-- we have unlimited space now on the-- those constraints that were [INAUDIBLE] constraints, we should really rethink those, and there were lots of conversations yesterday that I appreciated in the panels among what an online world looks like.
REBECCA KENNISON: But those constraints are really, really critical when what you have to do is choose who you're going to cite. And then-- and this is maybe where Josh will jump in. And then what is the context for those citations I think is really an important thing. And none of that is really captured in the way that metrics are done for the most part. It's not-- the narrative tends to be lost when all you're doing is listing what you're doing rather than why, how, who, and so on.
REBECCA KENNISON: So there needs to be more context.
SAI KONDA: Absolutely. Josh?
JOSH NICHOLSON: Yeah, so I would definitely second that we need more context. And so I think metrics are interesting because there's this tension between them being simplistic and not helping, but then being too simplistic. And so I think you can go away from metrics, but then you'll lose kind of the value of this as a heuristic. And so I think this tension will always kind of exist, and I think we're at an interesting stage where there's a diversity of metrics.
JOSH NICHOLSON: And I worry a little bit that we're getting to having kind of all these flare and badges for every different thing that we're all measuring. So open data for how it's been cited, supporting citations. And I think it could be a good thing, but I think we should be cautious about metrics for everything. I also really like what Rebecca said about the conversation happening amongst papers.
JOSH NICHOLSON: And so at scite, we show the citation context, in-text citation from the citing article, and I've started to really stop saying even citations. Say, look at this conversation that's happening amongst papers. And I think that is also really amazing and kind of where my love for citations is, because here is this conversation across generations, across disciplines that is preserved.
JOSH NICHOLSON: It's backed by data, it's backed by analysis, it's citeable, and I think that is really powerful, because you have this huge knowledge base that can be investigated and searched and utilized and better understood and valued and weighted and things like this. And so I'm excited to see the improvement of citations from scite, but also from others. And I am also a big fan of Garfield. I only got to speak with him on the phone.
JOSH NICHOLSON: But when I close my talks where I have slides, I generally show this PDF from 1964 where Eugene Garfield talked about adding more context to citations, and he called them citation markers. And I think it's been a challenge to do that because the scale of scholarly publishing is quite large. But I think with tools that have become available, we're now at a stage where small startups like ours can do really kind of innovative things.
JOSH NICHOLSON: And I think this will spread out into other areas, other types of data-- so not just looking only at scholarly publications, but alt metrics, I think there's room for improvement and things like this.
SAI KONDA: Thank you, Josh. Marie?
MARIE MCVEIGH: I think-- obviously we're a couple of Garfield fans here. But I think one of the things that I want to raise is the fact that citation itself was originally, as Rebecca said, it's individual and narrative of its own nature. It is-- we talk often in the selections group about the global scholarly dialogue. This is a conversation. So originally, citation metrics and the idea of citation indexing was based on the fact that Garfield saw value in this conversation that we could now explore at scale and also break some of the boundaries of time.
MARIE MCVEIGH: When you're reading a paper you go back and look-- you go back and look at the things that it referenced, but when you start to index and aggregate, then you can actually say this paper became these other things. And I think one of the insights that you're beginning to leverage in a new way is to say that information itself is plastic in that it changes as you use it.
MARIE MCVEIGH: The notion of citation indexing was based on the fact that if a paper is written, it can only contextualize itself based on what it knows up to that day. As the field evolves, as that paper is used and cited and discussed, then that paper acquires additional meanings which the index are at the time of-- or the author at the time of writing cannot predict and understand.
MARIE MCVEIGH: So that indexing actually is a way to draw information forward and continue to modify its meaning, show how it has evolved. So that was the underpinning of citation metric. The problem is, the awesome power of having 150 million of these highly meaningful interactions became a kind of statistical juggernaut and people stopped looking at that conversation, although the impetus was that conversation.
SAI KONDA: I think this is great, because I see context and narrative being a common theme in what all three of you have just spoken about. So why don't we just unpack that a little bit more. Just thinking about context, how do you think-- and also in theme of the New Directions webinar as well, how can we make best use of these metrics, number one, given the fact that, as Josh mentioned, we have a billion of them at this point? What is the best use for them?
SAI KONDA: And also, I would also like each of you to give your thoughts on what is the narrative here and how it can be applicable in that situation. Let's start with Rebecca, maybe.
REBECCA KENNISON: So I'd like to-- before I answer that question-- or kind of along with answering that question, I'd like to put in perhaps a bit of cautionary note. I agree with everything that Marie just said and that Josh said, but I am concerned about the people who aren't part of this conversation. So even if you have written a piece and it is now published and it's out there and we know that, again, thinking about humanity's work, some of this never gets cited at all.
REBECCA KENNISON: Does that mean that you're not part of the conversation? Does that mean that-- it certainly in a metrics way is a downside. Because what you need to do if you're an academic is report on not only what you published, but also how that's impacted your field, and the impact almost solely relies on who might be citing you.
REBECCA KENNISON: Except for when you have external review letters, like when you're going up for tenure or when you're going up for promotion, you get review letters where people talk about your work, and that might be the only time that some people talk about your work in that sense. And so while, again, I think the whole conversation is important, but if you're not being part of that conversation, I think that that can be a concern.
REBECCA KENNISON: I'm also concerned a bit about where work might be presented and how it might be presented in a way that is not part of the formal publication record. And I know that there are ways that people are looking at and capturing-- it's that kind of information. If you write a really well-regarded blog post-- or I don't know, a Twitter thread that is really used and could be a citeable conversation, how does that go in?
REBECCA KENNISON: And thinking more broadly about the metrics and the conversation and the narrative and everything. It's the full package of that conversation, it is not just within citations and not just within-- that scholarly conversation is only one part of the conversation. And all metrics, of course, have tried to try to look into uncovering some of that other conversations.
REBECCA KENNISON: But everything that I've just raised is also part of the metrics concern. Like some people really are very good at being out there and in social media and some people really aren't. Does that mean that you are not part of that conversation? And I'm not sure that I actually-- I kind of danced around to the question there, Sai, a bit, but I think that all of those things are really kind of important and I would like to hear my co-panelists unpack that a little bit more.
SAI KONDA: No, thank you, Rebecca. I think actually unpacked one thing that I was going to follow up in a few minutes later, is you mentioned impact. And I'm going to put that in quotes, and what we do really mean by impact. I think that's worth having a discussion right now since it's come up. Josh, Marie, if you have any thoughts on that.
JOSH NICHOLSON: Yeah. I think impact can mean so many things to many different people. I try not to use it. I think generally, though, to kind of come back to your question, most measures and metrics that we're looking at is looking at how many as opposed to how, and I think starting to see versus how many, whether that citations or tweets or mentions in policy documents, I think will start to provide more nuance and context and understanding.
JOSH NICHOLSON: And I think will start to allow us to use these metrics in new ways. So one thing that we see at scite is a lot of students using scite daily for their essay writing because they can see what have experts, what have PhDs or grad students written about this topic that I need to write about? And I would say they don't traditionally use bibliometric tools.
JOSH NICHOLSON: They're probably not going to Scopus or Web of Science to help them with an essay unless they're doing bibliometrics work. And so I think this data, whether that citations from scite or tweets from Twitter, can be used in new ways as we start to surface it in new ways, and I think that's particularly exciting because really, it is like people's writing, this conversation shouldn't just be boiled down to, here's a linkage, here's some impact, but guess what impact it is?
JOSH NICHOLSON: Or maybe don't even guess because it's too much work to kind of investigate that manually. Yeah.
SAI KONDA: Thank you, Josh. Marie?
MARIE MCVEIGH: I would love to see impact take the lesson of citation metrics, which is harvesting these small point-to-point interactions and saying, what can we do at scale to understand the landscape? And also, never erase that original aspect of that dialogue-- of that narrative. I'd say I love to-- I'm looking over here because I have the other screen where I pulled up the Merriam-Webster definition of impact and just lost the Zoom session.
MARIE MCVEIGH: And impinging or striking especially of one body against another, or a forceful contact or onset. Also the impetus communicated in and as if such a-- as if in such a context. I love that idea of this influence, this moment when you have changed your environment by the force of your idea, of your citation, of your presentation, of your panel. When you make people think in a different way, that's impact.
MARIE MCVEIGH: Now I would like to see that we could take this more generalized notion and say, how do we also bring this to a fact base that we can share? One metric being not perfect for everything doesn't mean no metric is perfect for anything. I wrote an essay a couple of years ago where I talk about the metaphor of impact or the metaphor and metrics where I talk about, let's look at thinking about metrics generally as a mosaic.
MARIE MCVEIGH: How do you take these individual things which don't capture the whole picture and assemble them so that you can look at something incredibly complicated, like academic production or academic performance? You don't use one tile for a mosaic, you use many, because the individual tile does not convey the meaning, but the assembly of them in a meaningful way is what contains that meaning.
MARIE MCVEIGH: So any metric has a context in which that metric itself can reflect some aspect of impact. But the picture as a whole is a much more nuanced thing.
SAI KONDA: I love your analogy about the mosaic and the tile view of looking at things. So I actually want to bring it back to where the impact-- where I see impact being used is the journal impact factor, obviously. And I think-- I want to hear all of your thoughts on this because as I see it, at the end of the day, it's a mathematical construct, it's a simple ratio, but then you translate that into-- and give it a term as a journal impact factor, it means a whole lot of things for different people.
SAI KONDA: So when especially when you talk about this mosaic picture and the journal impact factor being just one tile in that mosaic, how do we go about thinking about it, especially and also using the term context that all of you have mentioned before, what's the best way of really digesting this information whenever these numbers come out on an annual basis? And what it means, for example for humanities versus other areas?
SAI KONDA: What does it really mean for different areas?
REBECCA KENNISON: So the impact part-- I just want to-- that's a very nice lead-in, is if you know exactly what I'm going to say, which you don't, necessarily, which is that the time frame question, I think, is a really important one, and this is why I like the mosaic metaphor as well. Because impact factors are just like a three-year window, and in the humanities, of course that can be an extremely short window.
REBECCA KENNISON: It takes a very long time for ideas to percolate, and sometimes you don't see impact. But I'm also just thinking, because the Nobel Prizes are coming out, I'm reminded every year when the Nobel Prizes come out that those are like Lifetime Achievement Awards, if I can put it that way. They're for work that happened decades ago or decade-- or a decade ago.
REBECCA KENNISON: So even within, when you think about the impact of those particular people's work, it's only now that you're starting to really see that impact happening. I agree with Josh that I'm not sure impact is a word that kind of avoid a bit because I'm not sure-- I like the-- the dictionary definition, too. But impact on whom, by what.
REBECCA KENNISON: Because at the end of the day, we're talking about people doing-- having conversations and what is changing. And sometimes you can't capture that like I read a piece, and let's go back to citation practice. I read something but I don't publish a lot, but a lot of things influenced me. So that's an impact on me that's not being captured out there in the general conversation.
REBECCA KENNISON: But then there are also-- I'm thinking about Judith Butler's work, for example. Gender Trouble, which has been very transformative in the way that anybody thinks about gender construction. But that took 10 years before anybody really started to pay attention to that work. And now it's transformative, but along with everybody else. I mean-- and then the citation to Judith Butler happens, but not to all the colleagues who were also working within that context.
REBECCA KENNISON: So it's kind of a shorthand. So does that mean that Judith is more impactful than somebody else? Again, I'm troubling the waters here by having these-- by thinking about it this way, but again, who gets to be the marker for this impact I think is really important.
SAI KONDA: Thank you, Rebecca. Josh?
JOSH NICHOLSON: Yeah. So the impact factor and citation counts in general I think it's why I started scite. And so I think there is-- about six years ago, studies coming from large pharmaceutical companies and nonprofit initiatives looking at reproducibility in science. And I knew within my narrow field of chromosome segregation and aneuploidy which studies had been tested and either shown to be right or wrong.
JOSH NICHOLSON: But if someone else came into my field, you wouldn't know this information, and you'd see this highly cited paper with 300-plus citations. And so really, what we wanted to do initially and I think why we've started to do it is to say, hey, not all impact is kind of the same, and not all things that we're tracking, whether that's impact factor or total citation count or h-index, we're only looking at the quantity, not necessarily the quality.
JOSH NICHOLSON: And so wouldn't it be great if we could start to show more nuance, more understanding, more interpretation from other experts? And I really like, that's why it's this kind of continuation and it adds so much more even after publication. And so I think, as probably my-- I don't want to say hate, but like disliking of just how simple we use citations that led us to actually trying to improve upon that.
JOSH NICHOLSON: And it's kind of funny, because in that paper that I initially wrote saying, here's a better way that we could use citations, I suggest all these different groups should do it, including ISI and Thomson Reuters and ultimately that's not how things generally get done. And so we kind of stumbled into this. And I think what we've done now with citations I think ultimately will be done by all citation tools. And I see Clarivate already starting to show some citation context, and I think it's exciting to see that, because there's so much value in there that is lost on just a simple number.
JOSH NICHOLSON: And I think-- yeah. And that's true of impact factor we can hopefully--
MARIE MCVEIGH: I'm going to say, if you're not careful, you're going to start recommending that people actually read the paper.
JOSH NICHOLSON: Yeah.
SAI KONDA: Marie, any comments from you?
MARIE MCVEIGH: Only that there is-- we talk about the journal impact factor as though that were the only data point that we publish or the only metric that we publish. I did a calculation a long time ago-- and I will have to update it, but it gets a little intimidating, and that is at the time we published about 12,000 journal impact factors in that year. And I went and calculated how many data points were in the JCR that year, and it was over 8 million.
MARIE MCVEIGH: There is an incredible wealth and texture of data that we provide, and it's not two years long, it's 10 years long with indicators to any out-years further than that. We indicate that there is a greater depth of historical use of a journal through a metric that extends-- we had a cited half-life, the average age of materials in a journal that were cited in-- I think was the year 2017 data, that cited half-life of that journal was over 80 years, which means that 50% of the citations to that journal in one year were to materials that were more than 80 years old.
MARIE MCVEIGH: So it's not that we are not aware of that and that's not that we're not serving that up at a level of depth and texture. If all you're looking at is the one tile, you're not seeing even the mosaic that we're presenting in the one product where we contextualize that metric. It's not like, here's your impact factor, goodbye, go away.
MARIE MCVEIGH: It's like, no, here's where it came from, here's where it goes, here's what built it, here's the network of references. Rebecca with the ANH Journals in the JCR this year, there's a full citation network of cited and citing materials that also represents nonindexed materials. So everything is in there to create that context around, where is this journal in a year participating in the literature?
MARIE MCVEIGH: What does it reference, what reference is it? How do you start to get at where this individual item is placed in this broader context? Individual item being the journal and all of its content.
SAI KONDA: Thank you, Marie. All relevant points. I think there's a question that came in from the audience. And I think this is kind of related to what you just mentioned, and I wanted to put it out there in case others have thoughts on it. And the question is, as mentioned, impact factor is only one type of metric as we've been discussing, but many researchers and institutions laud it as the most significant and most important.
SAI KONDA: How can we better inform researchers to understand it is only one tile in the mosaic? So he said-- it's talking about the mechanism as to how we can educate people at this point.
MARIE MCVEIGH: Oh, golly. 25 years now I've been educating people. I still have people who don't understand [INAUDIBLE].. You said it was a ratio, bless your little heart, because it's been-- yeah, it's been called an average. So I think one of the terrible warnings to metrics that are coming down the pike is that if it's simple, it will be misunderstood and misused. If it's complicated, it won't be understood at all and it won't be used.
MARIE MCVEIGH: So how do you start to create this dialogue of the metric and its meaning? And I think that's really why I joined Clarivate, is to bring that in there and say, this-- OK, you have a number, but here it is in a five-year graph in the context of how this puts this journal in a category perspective. And here's all the articles and all the citations that created that number.
MARIE MCVEIGH: Oh, by the way, it's only 10% of the total citation activity. I look at the JCR as a way to look at a much more complicated picture of the value of a journal and that journal as a part of the global scholarly network and begin there and then use that. Go elsewhere from this. This is not one thing or one metric, this is one place to go for information, and then to take that much more nuanced view about, what are you looking to really value?
MARIE MCVEIGH: Do you need open access? Do you need timely turnaround? What is the real content of this journal and how does it perform? And how much is that impact factor based on one article with a large number of citations or a broad and deep-- basically a deep bench so that you can-- your journal is actually a continuously evolving set of valuable content?
MARIE MCVEIGH: So yeah, just-- this is a terrible thing, but look at the metric in the context in which we provide it. It's not by itself. We never do that.
REBECCA KENNISON: If I could jump in and say, I think that back behind that question is, why do people go to proxy-- why people go to proxy measures? And that is something that we all agree that it shouldn't. But it's not that people don't know that there are problems or so on. It's just that there's a vast amount that's being published and people are just trying to get a handle on, again, what is the-- yeah.
REBECCA KENNISON: What does any of this mean in terms of faculty productivity in particular? And so it becomes a question about who's going to get promoted and raises and so on, which is a very different kind of question than, what is the real impact? What is the real conversation that's happening and so on? And that goes back to the need for more narrative, more context, and reporting that out.
REBECCA KENNISON: But again, that then becomes a responsibility of the researcher. And then I want to stop and say, when we say the researcher, that's also what is being prized. It is the research side. But that's only one part of the enterprise, which has this three-legged stool of research and teaching and the big bucket of service where peer review and everything falls into that.
REBECCA KENNISON: And so one of the things that my colleague Chris Long, who's the Dean of the College of Arts and Letters at Michigan State, is doing is actually thinking differently about how he asks his faculty to describe the work that they're doing. So the work that they're doing, don't describe it as, what did you do in terms of research? What did you do in terms of teaching? What did you do in terms of service?
REBECCA KENNISON: But think about something like, how did you share knowledge this year? Now tell me how you shared knowledge. And that could be in your research and your publication. That could be in your teaching. That could be in your mentoring. That could be in the community conversations that you've had in your community engaged work. That could be anything.
REBECCA KENNISON: And it really has been very transformative for his faculty to think about things in that lens rather than how many pieces did you publish? How many classes did you teach? And how many articles to do review and so on. So it's a different way of looking at it, but I think that also helps-- if administrators would start to think about it that way, then those numbers are completely different.
REBECCA KENNISON: I mean, who cares about the journal impact factor within that context? It's not necessarily about sharing it.
SAI KONDA: Oh, that's a great point. Great initiative. And it almost takes me to the term that Josh mentioned in terms of conversations that's-- it's, how many conversations is your work kind of encouraging in the community? Josh, any closing comments on this particular topic?
JOSH NICHOLSON: Well, my neck was probably hurts from agreeing so much with what Marie said. Because I think-- the impact factor sticks around because it's simplistic and easy to understand, and we do need some of that. We're all overwhelmed with tons of things to do more than ever. And I think there have been probably way more sophisticated, much better metrics than the h-index, the impact factor. But oftentimes they're lost right.
JOSH NICHOLSON: And administrators and the people that are using these things, you have to think about, OK, who's the ones misusing it? Why are they misusing it? Can we make things more easy for people? There's this ideal world that we all want, but there's also reality, and you have to kind of go towards that. And I think acknowledge that we do need some proxies, we do need some heuristics to help these people, because it's impossible to sit around and read every single paper, watch every single talk, be inside someone's head.
JOSH NICHOLSON: And so looking at these things from kind of that viewpoint I think will help us design and kind of come up with better metrics and proxies, but I think there needs to be some realism in that. It's not enough to just say, we shouldn't have any of this, we should do all that because that's-- it just won't happen. And we use metrics outside of scholarly publishing for many things that we do and we appreciate that. And of course, there will be abuse of things.
JOSH NICHOLSON: We're all humans. There's abuse of everything. But I think-- I do think there's a lot of value in metrics to kind of round that back out. And so I really like this tension between simplistic and being adopted and way too sophisticated and no one ever uses. There needs to be somewhere in the middle, I think.
MARIE MCVEIGH: What about-- I don't know. One way that-- for good or ill-- and I'm really shooting from the hip here-- all metrics are in some degree a proxy and a part, not any individual. So I think we've got that kind of nailed amongst us. But what about the idea of metric as a possible filtration mechanism? So if I am interested in-- I'm looking for a journal to publish and I'm looking for journals in my field, I want to find the OA journals.
MARIE MCVEIGH: How do I find them or those journals that have a significant amount of this content? Well, there's a metric for that. And it's a way to lead in. It's not a way to nail down or close that door, it's a way to find an initial thing. And then you can dig deep into that context or into that situation. So how do you find a paper?
MARIE MCVEIGH: You can find it by a Google search, you can find it on Google Scholar, you can-- I don't know what their ranking is, but you can find it. You need to dig in to individually value that paper. How do you know amongst 3 million results you get when you search actin binding protein, how do I find the paper that's going to be relevant to me in this moment? And there's a lot of different mechanisms around that and a lot of different ways to start thinking in that direction.
MARIE MCVEIGH: So can we think of metrics as a filtration or as a way to identify outliers? This is something novel, this is something new. A metric of novelty is something everybody's been lusting after and unable to do because it requires time, and then it's not novel. So do we have-- do we have a way to move in that direction?
MARIE MCVEIGH: Josh, I'm really looking at you.
JOSH NICHOLSON: Sorry, I got distracted by the chat, and I was starting to read all those things. And so now I-- so I think-- I agree with kind of what you said, and I think how we get to where we're going and what we measure will be determined by kind of like, A, what is possible; and then B, what can be useful and also understandable? And so yeah.
JOSH NICHOLSON: I don't have too much to elaborate there because I got distracted, apologies.
MARIE MCVEIGH: I like that, what is possible and what is useful.
JOSH NICHOLSON: Yeah.
MARIE MCVEIGH: Because what is possible and what is useful is often a far cry from what is ideal and what we really want, but we should always be aiming towards that? What is ideal and what do we really want? So let's begin here and move in that direction. Yeah, I haven't looked at the chat and now you're making-- now you're making me scared.
JOSH NICHOLSON: Yeah.
SAI KONDA: Rebecca, you wanted to add something? Otherwise I was going to bring up the things from the chat.
REBECCA KENNISON: No. I just-- I was interested just to comment on the novelty question, which I think novelty and originality or interesting concepts when what we're talking about is that all of us are in a conversation with each other. And so I would just like to say I don't know how novel is any given idea when we're all building on the work of everybody else. But anyway, I'm just tossing that out there, as I think-- I'm not sure how one would assess the novelness or the originality of a particular work when the whole point is to build on everybody else's work.
REBECCA KENNISON: There. Sai, what do you want-- what do you want--
SAI KONDA: I think we can have a separate panel on the topic, I think. So those are very good questions that came in from the audience that I wanted to forward to you. So it's about, what about data citations? I think we spoke a lot about research citations. And are data citation tools like data citation index being used? And if so, how and by whom? Anyone would like to go first?
MARIE MCVEIGH: Data citation index, yes. We have been building that out for, what, 10, 12 years now? They started before I left, they were still at it. So we tried at that point to both identify and promulgate the idea that this is an important part of how this research is done. This is a necessary point of information. How do you recognize that? By creating a pointer to that. So the problem is in part the availability of an identifier.
MARIE MCVEIGH: And I will-- it might be the last Garfield quote, but it might not. In his 1955 essay, he actually says, oh, gee, wouldn't it be great if all the publishers would include this like code in the reference list that would indicate what this article is? And here's an idealized data extraction that it could be. And then everyone could do this group assembly of this stuff. I'm like, ooh.
MARIE MCVEIGH: DOI in 1955. So we needed a mechanism for data citation. We needed a way to identify that unique thing which is the entity cited in a data citation. So there's this kind of Necker cube of, well, you create an identifier for me first. Well then, I'll be able to cite it. Well then, you cite it so that I can create an identifier. But both of those things are progressing in parallel.
MARIE MCVEIGH: So I think we're moving in that direction. And again, once we get this at scale and a more generally adopted practice, we're going to be much further along. But the mechanism is starting to come into place. So there's an awful lot already there, and I think that we've laid the groundwork-- we as a community have laid that groundwork to start moving that forward more aggressively and more-- to create more use and more usefulness.
MARIE MCVEIGH:
JOSH NICHOLSON: Yeah. I mean, I guess I can add to this. So we-- this is a limitation of scite. So there are DOIs for many data sets from data side. There's different DOI registries, though, and there's the practical limitation of where can we get all this metadata from all these different areas? And so we don't do too much with data citations at least at scite.
JOSH NICHOLSON: But one thing that I can also mention, which I think is interesting, is we have this citation statement search. So you can search these fragments of text, extract it from the full text, and oftentimes this is used to say, I want to put in this URL for this data set to see how has this been used, because it's not in the reference list. And so I think there's still a lot of learning in the community to how-- where do I put my data, for one?
JOSH NICHOLSON: How do I make it referenceable? Like how do I then reference it myself in publications? And I think we're still pretty early there, at least from what my personal experience handling data and publishing data, which really kind of didn't happen much, to actually just using this search and seeing how other scientists reference it. They link to it, there's repositories, but it's still kind of treated differently.
JOSH NICHOLSON: And I think that will start to change. I don't know how quickly, though, and I think that comes to like, what's in it for me? There's a lot to do as a researcher or author now. This could be one more thing to maintain, one more thing to look at. And so you have to kind of align that as well and think about, OK, why is this person going to share their data?
JOSH NICHOLSON: Just for the good of the world? Is it going to benefit them? Do data citations count? Who's counting them? All these things. And so I think the groundwork is there, but I think we're still kind of in the infancy of looking at data citations, at least from my perspective.
MARIE MCVEIGH: It took 45 years to get a DOI. We had one internally. We're using internal codes, and it's funny, because that it had to be there, so Dr. G invented it. It just didn't get adopted.
REBECCA KENNISON: So when I was at PLOS a long time ago, when PLOS first started, we started by granularly identifying the idealized-- the graphics and then everything. Which was-- Crossref was like, we don't know how to handle this, but [INAUDIBLE], right? So to what Marie was saying, yeah, it's been a long time that we've been having like how do you persistently identify something that is data or-- But the problem has been, as you said, Josh, where is this data located?
REBECCA KENNISON: And also, again, going back to the context question, there are a lot of concerns particularly-- in this case, in social science about misinterpretation of the data, if it's just the data set without the context. And so in order to get people to give us data which we wanted in the repository at Columbia where I was working, we really did want data, I would tell them, well, give us your data and then give us everything-- all the stories that you've told about it in whatever format, wherever.
REBECCA KENNISON: And we'll put them all together and we'll package them together so that you can have the context. And that really did help, but I do think that there are some concerns just about what-- there's the persistence questions, like where is it and how can you find it? But then there's also the context question. And again, I think those things still need to be solved. In the same way as we've already been discussing about the whole record.
REBECCA KENNISON: There needs to be context, it needs to be findable, it needs to be persistent.
JOSH NICHOLSON: Persistent, I think, is an important one. Because-- so GitHub, where a lot of data lives, just introduced this ability that makes it easier to cite a GitHub repository. But GitHub is not, [INAUDIBLE]. I don't know the persistence of this. Maybe it is, maybe it's fine, but I think it doesn't come with a lot of these quality assurances that maybe less-used repos do. And so already there's some friction of like, OK, if I need to cite my data, can it be where I do all my work or I have to move it over to this other system that I don't use?
JOSH NICHOLSON: And so I think those things seem very small, but they matter a lot. And I think we get a lot of questions about scite and our machine learning and all this sophisticated stuff, but I think what is equally important-- and this ties into metrics in general, is how easy is it to use and understand? And I think that will be necessary for depositing data and citing data.
JOSH NICHOLSON: Because right now it's confusing. There's like this whole guideline of how you cite software as well, there's principles for it, even. And that friction will lead to like just the lack of adoption, I think, or slow the adoption.
SAI KONDA: I know we have seven minutes left on the clock. And so I want to maybe get one question in. Since DOI was mentioned multiple times, I thought this is a relevant one. So this is going back, there's a question from the audience on Rebecca's point on data mining, impactful citations. Citation metrics differ substantially across Google Scholar, Microsoft Academic, and Web of Science. Do we measure citations only by citing work as a DOI?
SAI KONDA: Would preprints be counted as valid citations? And what about grey literature?
JOSH NICHOLSON: So I-- yeah. So at scite, we use only DOI, and that's just a practical limitation. Like it's a great resource. There's 120 million records with metadata tied to it, but without DOI, scite probably would not exist. It would just be way too much work. And so there are other sources that are extremely valuable. Grey literature, I come from another company that I started called The Winnower, which was exclusively giving DOIs to gray literature.
JOSH NICHOLSON: So to blog posts, to Reddit science AMAs, all these things that didn't count because they weren't being cited. We tried to say, hey, this should count, and here's a way to make it easier to count it. And so I think there's practical limitations of harvesting all these things. It's not technically easy to do. There's not a lot of citation indexes out there because it's not something you just spin up and magically are grabbing information from all around the world.
JOSH NICHOLSON: It's Google Scholar, it's places that have been doing this for decades. And so I think those things do matter a lot. Preprints, I would also note, a lot of them have DOIs, a lot don't. And so I think this is something that is, when you're considering where you publish, you think about, OK, well people read this, and I think the same is true for kind of the data you use and the venues you put.
JOSH NICHOLSON: And personally, I probably wouldn't publish publishing something that didn't offer DOIs because I would know it would be harder to cite or could not be cited in some cases. But I do think there are new tools and efforts, and all kind of tools will start to explore this. Overton is one, that's looking at citations and policy documents.
JOSH NICHOLSON: And so starting to look at areas where we haven't-- it takes some effort, but it's definitely valuable and worth doing.
SAI KONDA: Thank you--
MARIE MCVEIGH: I'm running a reference search to see how many times The Winnower was cited in Web of Science. I've got two pages of versions of references, some with 15 citations.
JOSH NICHOLSON: Yeah.
MARIE MCVEIGH: That's the other bit of citation indexing. Is it's not about DOIs, which make it easy-- anybody can run a citation index on DOIs, but we were running a citation index for 45 years before there were DOIs. And it was about capturing the thing that is referenced in a metadata sense so that it can be unified, not that every occurrence-- no matter how unique the way that thing is referenced, oftentimes we are able to glom those together and say, this may be cited with a URL and then it cited without a URL, but we know that those are the same thing.
MARIE MCVEIGH: So we're probably looking at 200 or 300 citations in Web of Science that are trackable to-- and there were some with URLs and with-- I don't know if they have-- yep, some with DOIs, they're right there. So that idea of there is that visibility on that network and it has-- it's more than just a DOI, because not everything that's worth referencing and not everything that enters the scholarly literature as an item of discussion-- I always joke about Beyonce's Lemonade album as a feminist work, she doesn't have a DOI.
MARIE MCVEIGH: But she has a record in Web of Science as a cited work, because that is part of how the literature is evolving and using that work. Now I have to finish the search. I have to know how many times.
REBECCA KENNISON: Anything can have anything can have a DOI, but again, not everything gets indexed properly. And not to go down the metadata rabbit hole, but that's how-- when I was running the repository at Columbia-- not I-- my team was running the repository at Columbia, we spent a lot of time working with Google Scholar to optimize the metadata so that they would pick up everything that we have [INAUDIBLE]. And then we went and asked people for their blog posts and so on so that we could hold them in the repository so that they at least would have a [? DMI ?] and be citeable there.
REBECCA KENNISON: But working with everybody on what kind of metadata structures they need and so on I think is really, to Marie's point, the reason you can find Beyonce's album is because there is metadata about Beyonce's album. And the way that people cite it, there is a regularized kind of way that people can cite that. And so I don't want that to be the final word on anything, but to say thank you all the people who are working on metadata and metadata standards and enriching that metadata, we couldn't do it without you.
SAI KONDA: Thank you. And I think we will call this to a close with that note. I would like to-- great thanks to all three of you, Josh, Rebecca, and Marie, for very insightful comments. And thanks to the audience for sending all your wonderful questions. I personally learned a lot. I have a long list-- mental list of things to think about, one being to read more on Eugene Garfield's words.
SAI KONDA: Thank you, Marie, for mentioning. But I just wanted to say thanks to all of you. And just as a note to the audience, we have a 15-minute break following this session, after which there will be session on funds, funders, and funding. So please stay on, catch a break, and then jump back online. But thank you once again for your time, and hope everyone had a great fun listening to this panel.
SAI KONDA: And have a good rest of the day. Mary Beth, any closing comments?
MARY BETH BARILLA: No. Thanks, everyone, and we'll see you back in 15 minutes.
SAI KONDA: Thank you so much, everyone.
MARIE MCVEIGH: Thanks, everybody.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode