Cadmore media player playing video Alternative forms of research assessment and impact-NISO Plus
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
Alternative forms of research assessment and impact-NISO Plus
Description:
Alternative forms of research assessment and impact-NISO Plus
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/4024d5df-db05-4062-8afb-bc0843943af6/thumbnails/4024d5df-db05-4062-8afb-bc0843943af6.png?sv=2019-02-02&sr=c&sig=aKxkgjAP51N1FvLlji12DzBY6YSTM6ZkV76TrbjBytE%3D&st=2024-11-21T10%3A18%3A36Z&se=2024-11-21T14%3A23%3A36Z&sp=r
Duration:
T00H48M52S
Embed URL:
https://stream.cadmore.media/player/4024d5df-db05-4062-8afb-bc0843943af6
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/4024d5df-db05-4062-8afb-bc0843943af6/Alternative forms of research assessment and impact-NISO Plu.mp4?sv=2019-02-02&sr=c&sig=0uIAyIqcF3FxWPW3Mnb3MhprabX0l6T3mpEle2KbtqY%3D&st=2024-11-21T10%3A18%3A37Z&se=2024-11-21T12%3A23%3A37Z&sp=r
Upload Date:
2022-08-26T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
[MUSIC PLAYING]
SPEAKER: Hello, everyone. And welcome to NISO Plus 2022, the session that looks at alternative forms of research assessment and impact. Now, research is defined as the creation of new knowledge and/or the use of existing knowledge in new at creative ways so as to generate new concepts, methodologies, and understanding. Now, for most researchers, the continuous cycle of publishing papers and applying for funding is a mundane if accepted part of the job.
SPEAKER: But the over-reliance of metrics such as journal impact factor and other related metrics, too much importance is given to one small component of research. Thus, the pressure to find resources is incredibly competitive and fraught with frustration and disappointment as there are only a number of finite jobs or promotions or funds available. In today's session, our speakers will talk about some of the more meaningful metrics or alternative metrics that are being developed to measure the entire research and not just a component of it.
SPEAKER: Rachel will take us through some of the alternative forms of research evaluation and impact. She will then pass it on to Cameron and Lucy, who will talk about accountability and choice in selecting metrics for evaluation. And then lastly, Mike Taylor will tell us what big data and computation mean for research impact and assessment. So if research evaluation and impact is your jam, then this session is well worth your time.
SPEAKER: And if you're new to the field, don't worry at the end of the session, you'll know a lot more. And now, I'll pass it on to Rachel to kick this off.
RACHEL BORCHARDT: Thank you. I got to set up here. Thanks for having me today. So what I'm going to be talking about is three recent projects that I've been involved with that are really practical ways to take these, kind of push the boundaries of research evaluation and developing alternate indicators for impact. So we're going to just jump right in.
RACHEL BORCHARDT: The first one that I would like to talk about is that my university, American University, where we are currently in the process of revising tenure guidelines. This arose out of a task force report. And you can see on equity and promotion, and the original impetus for it was COVID, to really think about pushing the tenured deadline, rethinking people who weren't able to take sabbatical, things like that.
RACHEL BORCHARDT: And then kind of developed into more of a broader, how can we achieve a more equitable structure for tenure and promotion and increases in rank? So one of the big calls in the task force report was the incorporation of DEI values in all three areas of faculty life, scholarship, teaching, and service. So as part of that, the administrators compiled a bunch of resources to help every school and department, which is currently undergoing this process of updating their tenure guidelines.
RACHEL BORCHARDT: We've got some really big issues to tackle. How do you meaningfully incorporate DEI values into these three areas? So the library contributed two pieces of materials to start help faculty start thinking about how to go about this process. So we created short handouts, one just thinking about impact metrics and how to incorporate some different ideas, and then the other one was about open access and who we are as equitable models for teaching and scholarship.
RACHEL BORCHARDT: So as part of this process, I have been presenting thus far mostly about the impact metrics part to faculty. So linked there are slides from my most recent presentation, I'm going to summarize it. But if you'd like to see some more details, these have been fairly short presentations. So I really just try and hit highlights and then leave them with a really juicy bibliography where they can follow up if they're interested in additional details.
RACHEL BORCHARDT: So this is the main outline of what I try to cover to make the case for how we can incorporate DEI principles into impact metrics and evaluation. So part one, we really talk about the limits of existing metrics, right? And I try not to focus too much on impact factor but that is certainly one of the big ones I tried and centered on though maybe not explicitly. I'm talking about some of the limitations but more importantly, how some of these more traditional metrics tend to reinforce existing biases and power structures.
RACHEL BORCHARDT: And that's again, something that I let my scholarship section, the bibliography do a lot of the talking on my behalf, so that I'm not trying to convince faculty the entire time that there are biases but simply stating it so that we can move on. And also talking about how metrics have some limitations in what can be measured and what can be equitably compared. And the original task force report specifically called out interdisciplinary and cross-disciplinary research.
RACHEL BORCHARDT: So we talk about how metrics can often fail to really adequately capture contributions in those areas, especially when it comes to the journals that they're being published in. Part two is rethinking what impact is and especially getting away from this more narrow focus of scholarly impact into thinking about different impact audiences that aren't scholarly. And then I make the point that your metric should be driven by the impact audience.
RACHEL BORCHARDT: So I'm encouraging schools and departments to think about who are you trying to influence and how can we measure that influence? AU is a very applied social science university. Since we are in DC, as you can imagine policy and political figures are one of our potential impact audiences so I use that as an example. That's something like congressional testimony might be a measure of impact that you really would not capture if you're looking at something like impact factor or citation-based metrics.
RACHEL BORCHARDT: And then part three is the fact that metrics will always play some role but that there should be room for qualitative information as well. Just because you can measure it, doesn't mean that you necessarily should measure it. And I try and point out that there have been roles for qualitative information that are already pretty well entrenched especially for some disciplines such as humanities rely heavily on things like book reviews or talking about what your contribution to a research output is.
RACHEL BORCHARDT: These all provide a level of context and detail that a number or metric really can't provide. So that's the short introduction. So we'll move on to case two. This is another project that I worked on. And as part of the presentation to faculty, I actually incorporate this case, all of my cases are interwoven to some extent because it does demonstrate some of the principles I'm talking about of how you can name a specific non-scholarly audience, develop appropriate metrics or suggested metrics or measures for measuring impact to that audience and incorporate qualitative information as well.
RACHEL BORCHARDT: So it's called the framework for impactful Scholarship and metrics. It was created through a task force from ACRL, the Association of College and Research Libraries, and was approved in late 2020. It includes a wide range of different metrics and measures that are suggested. It is not a checklist, it's kind of a starting place for people to consider who they might be trying to influence and how to measure that impact.
RACHEL BORCHARDT: But in this case, we have two main categories of impact that we include in the framework. Scholarly or what we think of as more traditional impact measures and then practitioner as its own separate category and impact audience. This acknowledges that librarians tend to do research that is both informed by and informs our library practice as practitioners. So I thought it was important to really separate out those two because a lot of times we will have things like conference presentations, hello, that get watched and may get incorporated into practice but may not get cited.
RACHEL BORCHARDT: And one other important thing to note is that we do not include journal level metrics. There is an explanation for that. But especially for our field, there are some pretty easy points to make for why impact factor is not a particularly appropriate metric for our scholarship in our field. So with that, this is just a small snippet. It's about four pages long.
RACHEL BORCHARDT: We tried to be as inclusive as possible to include a very wide range of research outputs and some people may think of as practitioner outputs. For example, something like a lesson plan. You could find a space for it in our framework and think about different scholarly impact and practitioner impact metrics and measures. They'll notice quite a lot of these, especially when you get to the practitioner impact category are more qualitative in nature.
RACHEL BORCHARDT: They're a little less easy to define, but we felt they still had a place and an importance in describing it overall. Impact of a researcher or a department, et cetera. And finally case three, this one is, I would say the most speculative. It's still kind of a work in progress. The publication was actually published this week, though the research was a long time coming before it started as a way to just evaluate journals in our field, and we wanted to take a values-based approach to looking at how our journals are operating and how they are incorporating DEI values.
RACHEL BORCHARDT: So we ended up measuring three dimensions, openness, equity, and inclusion. We gathered information from those journals websites and also sent all of the editors a survey because a lot of these practices are not transparent. They may be happening behind the scenes but you wouldn't necessarily know that they had undergone bias training just from looking at their journal website.
RACHEL BORCHARDT: Even after the survey was completed, there's still a lot that we don't know, and trying to think through what is an ideal state and how can we tell when someone has achieved or is making progress towards a more ideal state of incorporation of DEI values. What is an ideal degree of commitment? How do we know that something is operationalized versus a project that's taken on by a single editor or editor-in-chief and editorial board at one moment in time?
RACHEL BORCHARDT: Are these long lasting effects or not? And as a result of this research, I took some of those same questions that we asked and turned it into a checklist for AU faculty which I will show you now. So this is the really speculative part. So please consider this a work in progress. But you can see for two of the values, openness and transparency and inclusive practice, it's essentially some ways to interrogate a journal if you are thinking about value-based publication decisions, right?
RACHEL BORCHARDT: I'm a researcher who wants to publish somewhere that incorporates these values, here are some ways to start to try and evaluate that. And again, recognizing that a lot of these are not easy to find. Even if I were to contact an editor, they may not be aware of all of these practices and we encountered that a lot in our research as well. I don't know if our back end is accessible, that's a really good question.
RACHEL BORCHARDT: Those kinds of things are feedback we receive from the survey quite often. So it's just kind of a start of a different way of thinking about journals that gets away from citation-based ways of evaluation. And with that, just if people have thoughts and would like to engage further outside of this presentation, please be in touch. I would be more than happy.
RACHEL BORCHARDT: To talk further, and now I'd like to pass it on to Cameron for their portion of the presentation.
CAMERON NEYLON: Thank you to speak. Great to follow up Rachel's presentation and to be able to talk about some of the work we're doing at Curtin University. I'm presenting on behalf of myself and my colleague, Professor Lucy Montgomery. One of the quantitative issues we face in the world is time zones.
CAMERON NEYLON: Our chair is up very late helping us with this session. But yeah, it's not always a challenge to be able to be available all times of the day in all the time zones. I want to talk about accountability for data and how building openness right the way through our systems can help to change the way we practice research in general. But I wanted to start by asking you how the weather is where you are, and to think about how you would tell me what the weather is like where you are.
CAMERON NEYLON: Because what you probably tell me is what the temperature is and then we'd have an argument about whether Fahrenheit or Celsius makes sense as a way to talk about temperatures. But you wouldn't necessarily think deeply about what you mean when you're talking about temperature. And indeed, it's really hard to imagine a world although this was true in the 15th and 16th century, well, we didn't have the concept of temperature because how could you imagine that the sort of damp heat of the tropics or the dry heat of Perth in Western Australia or a summer in San Francisco could be measured and quantified on a single numerical scale?
CAMERON NEYLON: That really doesn't make very much sense. And the concept of temperature as something that can be measured had to be invented, had to be developed. And once it was developed, we can now do things like talk about climate change, for instance, which will be really difficult to talk about what that means if we weren't able to talk about the difference between 1 and 1/2 degree rise and a 2 degree rise. But at the same time, we've lost a lot of the subtlety perhaps, about how we think about temperature, about climate.
CAMERON NEYLON: And in fact, what's kind of funny is that we have to reinvent some of these concepts. At the moment today in Bath it's cold and the feels like temperature is actually the same as the measured temperature, but also it isn't. So what I wanted to illustrate here, to talk through is the fact that when we try and understand something, when we care about something, we're interested in its behavior.
CAMERON NEYLON: We are particularly, as scientists, seek to quantify it. And when we can quantify it, then we can track it over time and we can measure it. But something really interesting happens there that the measure becomes our conception of the thing itself. And obviously, this is a metaphor for research evaluation, for when we think about what are the qualities of research that we care about, and then we turn that into account of citations or things like journal impact factors that have in some way become our conception of what research quality is, even though the concept itself, what we're trying to get at is much more subtle, much more complex, much more challenging.
CAMERON NEYLON: And of course, now, we could collect lots of big data and then we can track things in detail. So here's a tracking of a person. Clearly, if you look at these things and talk about where this is from or exactly what service provides it, but this is a form of data, a form of information based on pretty big data, this person is yeah, if you look at these things regularly, pretty mediocre.
CAMERON NEYLON: They seem to be some sort of biochemist or physicist, that they're doing some strange things along the way. And then we would assume that this is a way of representing a person, a researcher. Now of course, I'm showing this, this person is, in fact me. Here is another version of me from a different service, the one that a researcher will often go to because Google Scholar has bigger numbers than websites or Scopus.
CAMERON NEYLON: And indeed, my numbers here are bigger. So that's kind of good. But equally, I would say this is not a particularly good representation of my work, either in exactly the way that Rachel has been talking about. And here's another thing that's been doing the rounds this week on Twitter, a slightly unattributed but apparently from a training manual from IBM in the 1970s. The point being here, we can do a lot with computers, we can gather a lot of data, but at the end of the day, we have to make choices about the decisions that we make.
CAMERON NEYLON: And I think this notion that computers can't be held accountable and if the way in which we treat human beings is buried in an algorithm that can't be interrogated, can't be looked at, can't be criticized or critiqued, then we have a really quite serious problem. So that's nice in theory but what about the pragmatic realities that we're dealing with on a day-to-day basis?
CAMERON NEYLON: We have a changing environment, very rapidly changing environment over the last two years in particular. So what do we need to deal with? We're dealing with this big policy agendas around open science, around equity, diversity, and inclusion. And most of these are actually unlinked to any information capabilities that allow us to track whether we're making a difference or change things over time.
CAMERON NEYLON: Again, as Rachel was saying, that sense of when you ask a journal about their DEI activities, often they won't know, and if they do know, then that's again qualitative information which is difficult to track at scale. We have very strong demands for change. Often, changes for metrics. And we've already critiqued the journal impact factor several times already in this session.
CAMERON NEYLON: But have we made any suggestions about what it could be replaced with? What is people doing with it and how could that be changed? And we also have stretched staffing and skills capacities within institutions and organizations that don't necessarily well placed to deal with these needs right at the moment and right at this moment in time. So my argument, what I want to try and convince you of over the next five minutes or so as I rush through some examples is that open data systems can help to provide us collectively as a community with the capacity to answer better questions that respond to the needs of researchers and organizations, rather than doing what we often do at the moment, which is shape our questions to fit the available answers that the data can provide us with.
CAMERON NEYLON: To do that, I want to talk a little bit about building shared resources, and I want to very quickly introduce the work we're doing at Curtin through the curtain open knowledge initiative. The big picture about what we're trying to do at Curtin is a qualitative narrative-driven humanities project. We come from a school of creative arts, social inquiry, and media, and our goal is to change the stories that universities tell about themselves and place open knowledge at the heart of that narrative.
CAMERON NEYLON: We do a range of different things but a big part of what we do is to gather and integrate data from multiple sources and in particular, from open data sources that allow us to aggregate, re-use, and play and manipulate, and analyze that data. We believe we have one of the largest collections of this data. But there are a range of other people around the world doing this kind of integration and aggregation at the moment.
CAMERON NEYLON: And so what can we do with that? Well, we can do things like, for instance here is levels of open access over time in a particular country, doesn't particularly matter which country. We can do this scale, we can do this for any country, we can do this for any institution. That's all relatively easy. But what we could also do is change the way we think about open access and rather than imagining that open access through a repository only counts if that output is also not available through a journal.
CAMERON NEYLON: We can ask the question, what about all the things that are in repositories? We can ask different questions of the data. And that might seem like inside baseball, it might seem like a particularly nerdy kind of thing to worry about, and I've been obsessed about definitions of open access for longer than I care to admit to. But this is a real issue on the ground.
CAMERON NEYLON: In the week that we're recording these presentations. Plan S is reduced, released a report which seems to show that open access is going down over the last couple of years. The challenge they have in that report is that they don't actually have the data to make the claims and to measure the thing that they're actually trying to measure. And because the data is not shaped, not available in the way that they need to analyze their performance, they've actually generated a narrative which perhaps isn't as helpful and productive as it might be in understanding the kind of contribution they're making to the immediate access over time versus access, which is still encumbered by embargoes that exist in many of the data sets that we're dealing with.
CAMERON NEYLON: And this happens everywhere, we start to think about doing demos. What if you want to use a different benchmark to compare things to? What if you disagree with what's included or excluded? What if you want to check the process of the algorithm for errors? What if you want to improve the underlying data? And in our case, I can point you to the query, the code that does those open access assignments.
CAMERON NEYLON: And if you disagree with our open access assignments and our way of analyzing this data, you can write your own queries and your own systems to analyze and visualize that data differently. But that's still kind of conventional, right? So you can do a slightly different analysis of the same thing. What if you wanted to do something entirely different, and this is just an example of some work that my colleague, Dr. Karl Huang, has been doing recently, where we've taken an entirely different approach to looking at citations.
CAMERON NEYLON: And the question we've asked is not how many citations has an output got but what is the diversity of places which those citations come from? And the interesting thing as we're looking at this is that we see that for open access outputs, that the diversity of citation seems to be higher across the board. Now, I'm not asking you to trust me, I'm saying that link, you can go to a dashboard, you can play with the data that I generated this morning as I was preparing for this talk, and you can play around with that and think about it and see whether you agree or disagree with what I've done.
CAMERON NEYLON: You can look at some different measures as well as looking at different sets of outputs and also how that affects disciplines. There's also some other things going on that dashboard that you can dig into. The point I want to make is we analyzed 160 million different outputs, something like a billion citations, the whole data set, right? I didn't have to ask permission to do that.
CAMERON NEYLON: I didn't have to check with legal on whether there was some data licensing conditions that made it difficult, I didn't have to check with anyone before sharing that dashboard. The data is all open. And that's just the output side. There's also the opportunity, if we work collectively and together to improve these data sets because they're not perfect.
CAMERON NEYLON: There are many challenges with them and there's lots of improvements that can be done. But working collectively, there's the opportunity to improve the underlying data as well as the analysis and the tools we use to process it. So I come to a conclusion here and run through a whole bunch of things really very quickly. Evaluation, I've avoided talking about metrics to a large extent so a lot of people think more about evaluation.
CAMERON NEYLON: It's a moving target. It requires flexibility. But to do it effectively in a world which is changing and where we want to drive positive change in research practice and culture, we need control over the data and the systems and processes that make that possible. Right at the moment, I will make the assertion and I believe it to be true that open data sources, openly licensed data sources are as good or better or soon will be as the traditional proprietary data sources.
CAMERON NEYLON: We don't need to talk about the differences in coverage or completeness because we're getting to a point where those open data sources are just as good. I think really importantly, the benefits of scale and sharing that arise out of community and shared approaches to improving data make more sense in collective and community-based processes and organizations than they do in proprietary and closed systems. Services and analysis and intelligence and processing, these are all things that private organizations and proprietary approaches can work well on.
CAMERON NEYLON: But at scale of getting systems that give us data and flexible analysis for the kinds of things we actually care about, community approaches are not just better, they're actually going to be cheaper. So where I'm going to end with is really again coming back to this point. Do you and your teams want to be experts in reshaping questions, fit the available answers, or do you want to be the experts in helping researchers in institutions get answers to better questions?
CAMERON NEYLON: I'll finish there but just to say, if you're interested in more of the background of this, we have a book out. It is open access of course, via my M Press. And otherwise, I just need to thank a lot of people involved in this work. It is certainly not all work I've done. And the contribution the funding from a variety of sources. Most specifically, in fact, Curtin University which has supported this work very strongly.
CAMERON NEYLON: So with that, I will finish up and I will pass over to Mike.
MIKE TAYLOR: Thank you so much, Cameron. That was a really interesting talk. I really enjoyed both of the presentations so far. And I think given that we only had a relatively limited opportunity to communicate our ideas, I think that the three presentations together make a really nice matching set of talks. So I'm Mike Taylor, I'm a Digital Science. And I'm going to be talking about big data and computation, and what that means very much for the things that Cameron was just talking about, things that Rachel were talking about that increasing diversity and new ways of opening up and understanding the data that's available to us.
MIKE TAYLOR: And getting off that reliance on that one number to kill them all, one number to rule them all, not sure which one to go for. So I am going to start sharing at my screen hopefully. OK there we are. So hopefully you can see that. The presentation is called, what does big data and big computation mean for research impact and assessment?
MIKE TAYLOR: These are some absolutely enormous questions, which I'm going to try and deal with in about 10 minutes. This is me, I'm Mike Taylor. I've been at Digital Science for nearly six years, I was elsewhere for 20 years before that. I have a social science degree but I've always been involved in computation as you'll find up right from some very early days. I have just agreed a deadline for submitting a theses to my supervisor.
MIKE TAYLOR: So hopefully one day, my PhD will be complete. I expect Cameron and I have known about my research project since we both had much more hair and less of it was great. Oh, by the way, and if you want to grab hold of me, there is my email address digitalscience.com. I'm also on Twitter. So if you want to grab hold of me, tag me and ask me to talk about anything. I am one of those people who doesn't stop talking very easily.
MIKE TAYLOR: So I mentioned that I've been involved in computation for a very long time, not quite as long as the journal impact factor or using Garfield. Although when I started going to my secondary school in 1976, there was a computer science lab, it had a single terminal in it. It was dumb terminal, it was connected up to a mainframe somewhere, I don't know where. But in the middle of the computer lab, for some reason there was this cast iron device that could be used to punch holes in cards.
MIKE TAYLOR: It didn't look like this, it was much bigger than this. It was an absolute antique even in 1976. It wasn't being used by anybody. I guess it was being used by someone as a demonstrator model. It might be in a museum or would have been melted down. It was a massive device. But the thing that you're looking at is a card punch reader.
MIKE TAYLOR: And even now, when we people use Fortran in our computer language, it's still inherited some of the legacy from using columns to register pieces of data there. But I mentioned it because when we think about research assessment and calculation of metrics, the first time that anybody started calculating metrics like that was back in the 60s. Eugene Garfield, people of his generation back in the 60s, and it was taking months to calculate single numbers.
MIKE TAYLOR: And even then, they were using very, very simplified data sets, approaches which are still baked into the design of the general impact factor and other metrics. From now, we still are working with that legacy, which is decades old. But it took months to calculate those first data points. There used to be a blog that described it. Unfortunately, that blog seems to have disappeared.
MIKE TAYLOR: It's I can't find it anymore. A few years ago, 10 years ago, 12 years ago when I was working at Elsevier and we were working on the first iteration of the site space, sorry, the site score metric, we used to have to book time on not a mainframe set but a high performance computer. We would set up all the algorithms, we'd set up all the data. We would book time and set everything running on Friday evening and then go away for the weekend.
MIKE TAYLOR: Go out for beer and then wander home, come back into the office three or four days later. If our algorithms had failed, if they had failed, then we'd have to start again. And then I started Digital Science six years ago, and we were using a very early cloud-based thing. And I've put here it was a few seconds. Actually, it wasn't a few seconds. It was long enough making a cup of tea.
MIKE TAYLOR: So we've gone from decades in the 60s to 10, 12 years ago, taking a few days. A few years ago, it was taking long enough to make a cup of tea very important for this guy, long enough to wander off, make a cup of tea and come back again and see whether it worked or not. And now, I'm running those same calculations essentially free and in seconds. I had the bill for, my GDQ bill for December was, I think it was $0.42. It was an absurdly small amount of money that appeared on my credit card.
MIKE TAYLOR: So this is to give you an idea of what's happened to computational power and the ability to do those data. And I wanted to explore a little bit about what that's meant for how we can start pricing apart the data and getting some insights out of it. So as I say right at the very beginning and has been alluded to, we are coming out of a world where there was one number being used to rule them all, two if you include the H index.
MIKE TAYLOR: But these were relatively speaking straightforward calculations, that those general based metrics were very, very simplified to make them easy to calculate and to make it possible to calculate back in the 60s and 70s. And moving on, we've gone from having very simple number of data points to having many more data points even for doing things like the field weighted citation index, so article-based citation metrics or the field citation ratio, sorry, the relative citation ratio.
MIKE TAYLOR: Again, they're a little bit more complicated to calculate than a general one just in terms of the score, but they're still only based on that one single data source. There's no there's no diversity there. And as Cameron was just talking about, nowadays we have much more sophisticated needs, and as Rebecca was talking about, much more data that we can use to incorporate in our analysis. So the growth in computational power is really sparking our ability to do much faster, many more calculations, much more complexity.
MIKE TAYLOR: Coming back to the question that Cameron was raising about the answerability of the maths is a really interesting question. And I think that it's one of those things that certainly when I'm working with this maths, you have to think, how do I explain this? How do I put this in terms that it can be replicated? I'm not one of those people who believes in stuffing data into an environment and then crossing your fingers and hoping that whatever complex Bayesian calculations come back are actually an insight.
MIKE TAYLOR: You need to be able to track the data. And we as part of the academic world, we have that reliance both on research and openness and being able to interrogate each other. So just thinking about that shift from small data to big data. And as I mentioned, general metrics were very much simplified to reduce the number of data points to be able to put them onto those-- to be able to define them using a holes in cards and to compute them in Philadelphia in the 60s.
MIKE TAYLOR: So to calculate a general level citation metric really involves very, very few number of data points. Relatively speaking, this is quite an easy thing to calculate, quite cheap to calculate. And if we were going to do the same kind of approach, a very simple approach, then we're talking about perhaps between 40 and 40 million data points to calculate that for almetrics. And that's not even doing anything particularly interesting or complicated with the data.
MIKE TAYLOR: So for example, when I'm running the calculations that I was talking about in December, I'm adding in multiple levels of classification, many, many more data points to run it. So not thinking about 60,000 data points, not thinking about four million data points or even 40 million data points, but rather by the time we start taking all of these pieces of data into account over multiple years to try and calculate the dynamics of, we easily realize that we are calculating in excess of a billion data points over the course of this.
MIKE TAYLOR: And the reason we want to do this is because we want to answer some of those questions that the Cameron's raising. Some of those more complicated questions about dynamics and how a policy in publication funding for example, how is that affecting the downstream impact of your research? Because of course, as Rebecca has said, impact is not a single thing. We're not talking about simple things like citations, rather than we're thinking about public engagement, we're thinking about economic impact, we're thinking more about diversity, who is citing us?
MIKE TAYLOR: Who's reading us? Who is engaging in our research? And do that. We now have these platforms, we have access to the data, we have access to the computational power, and it literally takes seconds. I cannot make a cup of tea when I'm running this calculation. This calculation that exceeds over a billion data points, when I calculate it, it is taking seconds.
MIKE TAYLOR: I have a barely enough time to take a swig of water, not a time to make a cup of tea or go on holiday as it might have been in the 60s or 70s. So this computational power, the access to the data. The fact that so much of this data is open and available to us to compute, it means that we can start answering much more complicated questions. But to do this requires something of a shift in mindset.
MIKE TAYLOR: Because we seem to have evolved out of this world where we had this one single number, one set of numbers to answer questions. But we started trying to answer many, many questions using the same numbers. And this was a consequence of, what was the only number we have. We only had these single numbers. So we were trying to use the same tool to answer a number of different sets.
MIKE TAYLOR: This world that we're living in is much more complicated. And I think that we've seen that there's quite a lot of resistance to change. Some of those approaches that we've been talking about that go back years and decades have been very robust, they've been criticized, quite rightly criticized, but there's a reluctance to change in the world. As I'll mention it, it's very variable whereabouts that reluctance, whereabouts to that reluctance is firm.
MIKE TAYLOR: But questions like, what does excellent research mean? They are really hard-- they are really hard to answer partly because we try to express questions that could be answered in the data that we used to have to use. And the world of big data and big computation, faster computation, means that we really need to step back from the problem and start really addressing those really addressing ourselves to the questions that we are able to answer.
MIKE TAYLOR: So rather than asking quite primitive questions, if you like, of how much excellent research, we can start defining what excellent means in terms of the data that can answer those questions. So that primitive question, how much excellent research can you publish? We can start asking much more complicated questions like, how much excellent-- in terms of that, what excellent research means.
MIKE TAYLOR: Does it mean more diverse? Does it mean we want to be cited by the Lancet in the New England Journal of Medicine? Does it mean that we want to have a broad, a diverse? I've got a really, really nice example of one of those. So I did some work for an academic institution in the US. And they were using what I would say without mentioning who they are, and I'm not going to mention who they are because they've sort of moved on.
MIKE TAYLOR: But two or three years ago, the question that they were asking themselves to understand how much excellent research was very much based on whether or not their academics were publishing in the New England Journal of Medicine, the Lancet. It was fairly, that's fairly simple because they had a certain threshold using a journal metric and excellent was a line that they sort of arbitrarily drew around that point.
MIKE TAYLOR: Now, when I came in, we were having a look at some of their data and trying to understand what was excellent. And to Cameron's point about diversity, we were able to identify one of their researchers who had never gone anywhere near the New England Journal of Medicine or the Lancet, and was publishing in open journals. I think it was Post-Tropical Medicine.
MIKE TAYLOR: But her research was having an enormous impact globally. And you were looking at the diversity of the citations, looking at the diversity of health metrics. And from my perspective, my argument was, these are the people you should be celebrating because the impact they're having is measurably much more than someone whose article may or may not be getting any citations but was in the New England Journal of Medicine.
MIKE TAYLOR: So the new world, the old world, very arbitrary thresholds. The new world, much more complicated, much more sophisticated. And really, we have the ability to answer questions which are much more precisely defined, much more attuned to the data that we have, and it's much faster for us to explore these answers. We don't have to make these big investments and huge clunky systems, rather we can be agile, we can be interrogative, we can form hypotheses and we can test those hypotheses using data that anyone can run.
MIKE TAYLOR: I did AI did a very quick little example because I thought this was quite fun. So I took a set of three different journals that look as if they're the same sort of journal and plotted out some data. So I ran this data just for this presentation. So back in the old days, we would have had one citation-based metric to compare these three journals.
MIKE TAYLOR: Now, we've got a world of data that we can compute to understand much more about the performance of an individual journal. So these are-- sorry, I left a line in there from a previous slide. So these are three journals, International Dairy Journal, Journal of Dairy Science, Journal of Dairy Research. So they look very similar to each other on the surface. They sort of publish science from more or less the same areas.
MIKE TAYLOR: International Dairy Journal appears to be slightly more technical, slightly less veterinary. JDS publishes about 900 articles a year. Journal of Dairy Research is a little bit smaller but nevertheless, they're more or less all in the same space. And I thought it'd be fun to have a look to see what big data can reveal. So really nice example here is sort of patterns where we can clearly see that the International Dairy Journal gets far more patent activity around it.
MIKE TAYLOR: Interestingly enough, if you start breaking out how well it's agricultural science papers perform compared to the other, you start getting some different things. But this is data that's being driven from volumetric data, it's being driven from dimensions, it's being normalized by subject areas on a very fractional basis. And we can see that there's a distinct difference between those two journals.
MIKE TAYLOR: If we have a look at the news. So the relatively the relative standing of the news coverage. So in this case, the Journal of Dairy Science was this medium lower journal for getting news coverage between 2010, 2014. Something seems to happen in 2015 because subsequent from 2015, that journal gets a lot more news attention than either of the two journals. I get looking very, very similar.
MIKE TAYLOR: This is computed over all of those data points, millions of data points. Tweets, really, really interesting. And again, this one journal, Journal of Dairy Science seems to have gone, seems to have been turbocharged in 2014, an enormous uptick in the amount of Twitter coverage from it. And again, these aren't counts, these are normalized balanced by all of the different subject areas.
MIKE TAYLOR: And I could break this down by diversity, I could break this down by open-access status, I could look at it in all different ways. The one thing I can absolutely say, this has got nothing to do with the rate of citations. So that ordinary metric, if we have a look at the ordinary metric. In this case, we can see that the Journal of Dairy Science, the one that's been showing that enormous uptick in news coverage and Twitter coverage, nothing to do with the academic standing of it.
MIKE TAYLOR: The citations for each one of those journals apart from the last year is relatively static. So we've gone through several stages over the course of the decades. We've gone from having legacy metrics, metrics that were based on old ways of doing computer closed data, very, very hard, very expensive. Lots of investment had to go in to calculate those numbers. And we've moving away from that to a world where we've got open data, where we've got fast data, where we've got big data, where we can do all of this hypothesis testing using data science approaches, where we can shape the questions and come up with more agile answers.
MIKE TAYLOR: Now, my question I suppose that I've left thinking is, well, is the academic research evaluation process lagging behind the data, behind the data science? I think it is. I do a lot of work with organizations that do research from all around the world and in many, many different sectors. And it suggests to me that the corporates are much more willing to accept a data science approach, this data approach. They're much more interested in agile approaches to ask questions about research and answer questions about it.
MIKE TAYLOR: Whereas, absolutely in academia, we do need to be a bit more cautious, a little bit more careful about how we develop. I'm still looking forward to the next five, 10 years. I think is going to be really exciting for research assessment. And with that, I'm going to stop and hand back to Melroy.
SPEAKER: Thank you very much for that Rachel, Mike, and Cameron. [MUSIC PLAYING]
Segment:0 .
[MUSIC PLAYING]
SPEAKER: Hello, everyone. And welcome to NISO Plus 2022, the session that looks at alternative forms of research assessment and impact. Now, research is defined as the creation of new knowledge and/or the use of existing knowledge in new at creative ways so as to generate new concepts, methodologies, and understanding. Now, for most researchers, the continuous cycle of publishing papers and applying for funding is a mundane if accepted part of the job.
SPEAKER: But the over-reliance of metrics such as journal impact factor and other related metrics, too much importance is given to one small component of research. Thus, the pressure to find resources is incredibly competitive and fraught with frustration and disappointment as there are only a number of finite jobs or promotions or funds available. In today's session, our speakers will talk about some of the more meaningful metrics or alternative metrics that are being developed to measure the entire research and not just a component of it.
SPEAKER: Rachel will take us through some of the alternative forms of research evaluation and impact. She will then pass it on to Cameron and Lucy, who will talk about accountability and choice in selecting metrics for evaluation. And then lastly, Mike Taylor will tell us what big data and computation mean for research impact and assessment. So if research evaluation and impact is your jam, then this session is well worth your time.
SPEAKER: And if you're new to the field, don't worry at the end of the session, you'll know a lot more. And now, I'll pass it on to Rachel to kick this off.
RACHEL BORCHARDT: Thank you. I got to set up here. Thanks for having me today. So what I'm going to be talking about is three recent projects that I've been involved with that are really practical ways to take these, kind of push the boundaries of research evaluation and developing alternate indicators for impact. So we're going to just jump right in.
RACHEL BORCHARDT: The first one that I would like to talk about is that my university, American University, where we are currently in the process of revising tenure guidelines. This arose out of a task force report. And you can see on equity and promotion, and the original impetus for it was COVID, to really think about pushing the tenured deadline, rethinking people who weren't able to take sabbatical, things like that.
RACHEL BORCHARDT: And then kind of developed into more of a broader, how can we achieve a more equitable structure for tenure and promotion and increases in rank? So one of the big calls in the task force report was the incorporation of DEI values in all three areas of faculty life, scholarship, teaching, and service. So as part of that, the administrators compiled a bunch of resources to help every school and department, which is currently undergoing this process of updating their tenure guidelines.
RACHEL BORCHARDT: We've got some really big issues to tackle. How do you meaningfully incorporate DEI values into these three areas? So the library contributed two pieces of materials to start help faculty start thinking about how to go about this process. So we created short handouts, one just thinking about impact metrics and how to incorporate some different ideas, and then the other one was about open access and who we are as equitable models for teaching and scholarship.
RACHEL BORCHARDT: So as part of this process, I have been presenting thus far mostly about the impact metrics part to faculty. So linked there are slides from my most recent presentation, I'm going to summarize it. But if you'd like to see some more details, these have been fairly short presentations. So I really just try and hit highlights and then leave them with a really juicy bibliography where they can follow up if they're interested in additional details.
RACHEL BORCHARDT: So this is the main outline of what I try to cover to make the case for how we can incorporate DEI principles into impact metrics and evaluation. So part one, we really talk about the limits of existing metrics, right? And I try not to focus too much on impact factor but that is certainly one of the big ones I tried and centered on though maybe not explicitly. I'm talking about some of the limitations but more importantly, how some of these more traditional metrics tend to reinforce existing biases and power structures.
RACHEL BORCHARDT: And that's again, something that I let my scholarship section, the bibliography do a lot of the talking on my behalf, so that I'm not trying to convince faculty the entire time that there are biases but simply stating it so that we can move on. And also talking about how metrics have some limitations in what can be measured and what can be equitably compared. And the original task force report specifically called out interdisciplinary and cross-disciplinary research.
RACHEL BORCHARDT: So we talk about how metrics can often fail to really adequately capture contributions in those areas, especially when it comes to the journals that they're being published in. Part two is rethinking what impact is and especially getting away from this more narrow focus of scholarly impact into thinking about different impact audiences that aren't scholarly. And then I make the point that your metric should be driven by the impact audience.
RACHEL BORCHARDT: So I'm encouraging schools and departments to think about who are you trying to influence and how can we measure that influence? AU is a very applied social science university. Since we are in DC, as you can imagine policy and political figures are one of our potential impact audiences so I use that as an example. That's something like congressional testimony might be a measure of impact that you really would not capture if you're looking at something like impact factor or citation-based metrics.
RACHEL BORCHARDT: And then part three is the fact that metrics will always play some role but that there should be room for qualitative information as well. Just because you can measure it, doesn't mean that you necessarily should measure it. And I try and point out that there have been roles for qualitative information that are already pretty well entrenched especially for some disciplines such as humanities rely heavily on things like book reviews or talking about what your contribution to a research output is.
RACHEL BORCHARDT: These all provide a level of context and detail that a number or metric really can't provide. So that's the short introduction. So we'll move on to case two. This is another project that I worked on. And as part of the presentation to faculty, I actually incorporate this case, all of my cases are interwoven to some extent because it does demonstrate some of the principles I'm talking about of how you can name a specific non-scholarly audience, develop appropriate metrics or suggested metrics or measures for measuring impact to that audience and incorporate qualitative information as well.
RACHEL BORCHARDT: So it's called the framework for impactful Scholarship and metrics. It was created through a task force from ACRL, the Association of College and Research Libraries, and was approved in late 2020. It includes a wide range of different metrics and measures that are suggested. It is not a checklist, it's kind of a starting place for people to consider who they might be trying to influence and how to measure that impact.
RACHEL BORCHARDT: But in this case, we have two main categories of impact that we include in the framework. Scholarly or what we think of as more traditional impact measures and then practitioner as its own separate category and impact audience. This acknowledges that librarians tend to do research that is both informed by and informs our library practice as practitioners. So I thought it was important to really separate out those two because a lot of times we will have things like conference presentations, hello, that get watched and may get incorporated into practice but may not get cited.
RACHEL BORCHARDT: And one other important thing to note is that we do not include journal level metrics. There is an explanation for that. But especially for our field, there are some pretty easy points to make for why impact factor is not a particularly appropriate metric for our scholarship in our field. So with that, this is just a small snippet. It's about four pages long.
RACHEL BORCHARDT: We tried to be as inclusive as possible to include a very wide range of research outputs and some people may think of as practitioner outputs. For example, something like a lesson plan. You could find a space for it in our framework and think about different scholarly impact and practitioner impact metrics and measures. They'll notice quite a lot of these, especially when you get to the practitioner impact category are more qualitative in nature.
RACHEL BORCHARDT: They're a little less easy to define, but we felt they still had a place and an importance in describing it overall. Impact of a researcher or a department, et cetera. And finally case three, this one is, I would say the most speculative. It's still kind of a work in progress. The publication was actually published this week, though the research was a long time coming before it started as a way to just evaluate journals in our field, and we wanted to take a values-based approach to looking at how our journals are operating and how they are incorporating DEI values.
RACHEL BORCHARDT: So we ended up measuring three dimensions, openness, equity, and inclusion. We gathered information from those journals websites and also sent all of the editors a survey because a lot of these practices are not transparent. They may be happening behind the scenes but you wouldn't necessarily know that they had undergone bias training just from looking at their journal website.
RACHEL BORCHARDT: Even after the survey was completed, there's still a lot that we don't know, and trying to think through what is an ideal state and how can we tell when someone has achieved or is making progress towards a more ideal state of incorporation of DEI values. What is an ideal degree of commitment? How do we know that something is operationalized versus a project that's taken on by a single editor or editor-in-chief and editorial board at one moment in time?
RACHEL BORCHARDT: Are these long lasting effects or not? And as a result of this research, I took some of those same questions that we asked and turned it into a checklist for AU faculty which I will show you now. So this is the really speculative part. So please consider this a work in progress. But you can see for two of the values, openness and transparency and inclusive practice, it's essentially some ways to interrogate a journal if you are thinking about value-based publication decisions, right?
RACHEL BORCHARDT: I'm a researcher who wants to publish somewhere that incorporates these values, here are some ways to start to try and evaluate that. And again, recognizing that a lot of these are not easy to find. Even if I were to contact an editor, they may not be aware of all of these practices and we encountered that a lot in our research as well. I don't know if our back end is accessible, that's a really good question.
RACHEL BORCHARDT: Those kinds of things are feedback we receive from the survey quite often. So it's just kind of a start of a different way of thinking about journals that gets away from citation-based ways of evaluation. And with that, just if people have thoughts and would like to engage further outside of this presentation, please be in touch. I would be more than happy.
RACHEL BORCHARDT: To talk further, and now I'd like to pass it on to Cameron for their portion of the presentation.
CAMERON NEYLON: Thank you to speak. Great to follow up Rachel's presentation and to be able to talk about some of the work we're doing at Curtin University. I'm presenting on behalf of myself and my colleague, Professor Lucy Montgomery. One of the quantitative issues we face in the world is time zones.
CAMERON NEYLON: Our chair is up very late helping us with this session. But yeah, it's not always a challenge to be able to be available all times of the day in all the time zones. I want to talk about accountability for data and how building openness right the way through our systems can help to change the way we practice research in general. But I wanted to start by asking you how the weather is where you are, and to think about how you would tell me what the weather is like where you are.
CAMERON NEYLON: Because what you probably tell me is what the temperature is and then we'd have an argument about whether Fahrenheit or Celsius makes sense as a way to talk about temperatures. But you wouldn't necessarily think deeply about what you mean when you're talking about temperature. And indeed, it's really hard to imagine a world although this was true in the 15th and 16th century, well, we didn't have the concept of temperature because how could you imagine that the sort of damp heat of the tropics or the dry heat of Perth in Western Australia or a summer in San Francisco could be measured and quantified on a single numerical scale?
CAMERON NEYLON: That really doesn't make very much sense. And the concept of temperature as something that can be measured had to be invented, had to be developed. And once it was developed, we can now do things like talk about climate change, for instance, which will be really difficult to talk about what that means if we weren't able to talk about the difference between 1 and 1/2 degree rise and a 2 degree rise. But at the same time, we've lost a lot of the subtlety perhaps, about how we think about temperature, about climate.
CAMERON NEYLON: And in fact, what's kind of funny is that we have to reinvent some of these concepts. At the moment today in Bath it's cold and the feels like temperature is actually the same as the measured temperature, but also it isn't. So what I wanted to illustrate here, to talk through is the fact that when we try and understand something, when we care about something, we're interested in its behavior.
CAMERON NEYLON: We are particularly, as scientists, seek to quantify it. And when we can quantify it, then we can track it over time and we can measure it. But something really interesting happens there that the measure becomes our conception of the thing itself. And obviously, this is a metaphor for research evaluation, for when we think about what are the qualities of research that we care about, and then we turn that into account of citations or things like journal impact factors that have in some way become our conception of what research quality is, even though the concept itself, what we're trying to get at is much more subtle, much more complex, much more challenging.
CAMERON NEYLON: And of course, now, we could collect lots of big data and then we can track things in detail. So here's a tracking of a person. Clearly, if you look at these things and talk about where this is from or exactly what service provides it, but this is a form of data, a form of information based on pretty big data, this person is yeah, if you look at these things regularly, pretty mediocre.
CAMERON NEYLON: They seem to be some sort of biochemist or physicist, that they're doing some strange things along the way. And then we would assume that this is a way of representing a person, a researcher. Now of course, I'm showing this, this person is, in fact me. Here is another version of me from a different service, the one that a researcher will often go to because Google Scholar has bigger numbers than websites or Scopus.
CAMERON NEYLON: And indeed, my numbers here are bigger. So that's kind of good. But equally, I would say this is not a particularly good representation of my work, either in exactly the way that Rachel has been talking about. And here's another thing that's been doing the rounds this week on Twitter, a slightly unattributed but apparently from a training manual from IBM in the 1970s. The point being here, we can do a lot with computers, we can gather a lot of data, but at the end of the day, we have to make choices about the decisions that we make.
CAMERON NEYLON: And I think this notion that computers can't be held accountable and if the way in which we treat human beings is buried in an algorithm that can't be interrogated, can't be looked at, can't be criticized or critiqued, then we have a really quite serious problem. So that's nice in theory but what about the pragmatic realities that we're dealing with on a day-to-day basis?
CAMERON NEYLON: We have a changing environment, very rapidly changing environment over the last two years in particular. So what do we need to deal with? We're dealing with this big policy agendas around open science, around equity, diversity, and inclusion. And most of these are actually unlinked to any information capabilities that allow us to track whether we're making a difference or change things over time.
CAMERON NEYLON: Again, as Rachel was saying, that sense of when you ask a journal about their DEI activities, often they won't know, and if they do know, then that's again qualitative information which is difficult to track at scale. We have very strong demands for change. Often, changes for metrics. And we've already critiqued the journal impact factor several times already in this session.
CAMERON NEYLON: But have we made any suggestions about what it could be replaced with? What is people doing with it and how could that be changed? And we also have stretched staffing and skills capacities within institutions and organizations that don't necessarily well placed to deal with these needs right at the moment and right at this moment in time. So my argument, what I want to try and convince you of over the next five minutes or so as I rush through some examples is that open data systems can help to provide us collectively as a community with the capacity to answer better questions that respond to the needs of researchers and organizations, rather than doing what we often do at the moment, which is shape our questions to fit the available answers that the data can provide us with.
CAMERON NEYLON: To do that, I want to talk a little bit about building shared resources, and I want to very quickly introduce the work we're doing at Curtin through the curtain open knowledge initiative. The big picture about what we're trying to do at Curtin is a qualitative narrative-driven humanities project. We come from a school of creative arts, social inquiry, and media, and our goal is to change the stories that universities tell about themselves and place open knowledge at the heart of that narrative.
CAMERON NEYLON: We do a range of different things but a big part of what we do is to gather and integrate data from multiple sources and in particular, from open data sources that allow us to aggregate, re-use, and play and manipulate, and analyze that data. We believe we have one of the largest collections of this data. But there are a range of other people around the world doing this kind of integration and aggregation at the moment.
CAMERON NEYLON: And so what can we do with that? Well, we can do things like, for instance here is levels of open access over time in a particular country, doesn't particularly matter which country. We can do this scale, we can do this for any country, we can do this for any institution. That's all relatively easy. But what we could also do is change the way we think about open access and rather than imagining that open access through a repository only counts if that output is also not available through a journal.
CAMERON NEYLON: We can ask the question, what about all the things that are in repositories? We can ask different questions of the data. And that might seem like inside baseball, it might seem like a particularly nerdy kind of thing to worry about, and I've been obsessed about definitions of open access for longer than I care to admit to. But this is a real issue on the ground.
CAMERON NEYLON: In the week that we're recording these presentations. Plan S is reduced, released a report which seems to show that open access is going down over the last couple of years. The challenge they have in that report is that they don't actually have the data to make the claims and to measure the thing that they're actually trying to measure. And because the data is not shaped, not available in the way that they need to analyze their performance, they've actually generated a narrative which perhaps isn't as helpful and productive as it might be in understanding the kind of contribution they're making to the immediate access over time versus access, which is still encumbered by embargoes that exist in many of the data sets that we're dealing with.
CAMERON NEYLON: And this happens everywhere, we start to think about doing demos. What if you want to use a different benchmark to compare things to? What if you disagree with what's included or excluded? What if you want to check the process of the algorithm for errors? What if you want to improve the underlying data? And in our case, I can point you to the query, the code that does those open access assignments.
CAMERON NEYLON: And if you disagree with our open access assignments and our way of analyzing this data, you can write your own queries and your own systems to analyze and visualize that data differently. But that's still kind of conventional, right? So you can do a slightly different analysis of the same thing. What if you wanted to do something entirely different, and this is just an example of some work that my colleague, Dr. Karl Huang, has been doing recently, where we've taken an entirely different approach to looking at citations.
CAMERON NEYLON: And the question we've asked is not how many citations has an output got but what is the diversity of places which those citations come from? And the interesting thing as we're looking at this is that we see that for open access outputs, that the diversity of citation seems to be higher across the board. Now, I'm not asking you to trust me, I'm saying that link, you can go to a dashboard, you can play with the data that I generated this morning as I was preparing for this talk, and you can play around with that and think about it and see whether you agree or disagree with what I've done.
CAMERON NEYLON: You can look at some different measures as well as looking at different sets of outputs and also how that affects disciplines. There's also some other things going on that dashboard that you can dig into. The point I want to make is we analyzed 160 million different outputs, something like a billion citations, the whole data set, right? I didn't have to ask permission to do that.
CAMERON NEYLON: I didn't have to check with legal on whether there was some data licensing conditions that made it difficult, I didn't have to check with anyone before sharing that dashboard. The data is all open. And that's just the output side. There's also the opportunity, if we work collectively and together to improve these data sets because they're not perfect.
CAMERON NEYLON: There are many challenges with them and there's lots of improvements that can be done. But working collectively, there's the opportunity to improve the underlying data as well as the analysis and the tools we use to process it. So I come to a conclusion here and run through a whole bunch of things really very quickly. Evaluation, I've avoided talking about metrics to a large extent so a lot of people think more about evaluation.
CAMERON NEYLON: It's a moving target. It requires flexibility. But to do it effectively in a world which is changing and where we want to drive positive change in research practice and culture, we need control over the data and the systems and processes that make that possible. Right at the moment, I will make the assertion and I believe it to be true that open data sources, openly licensed data sources are as good or better or soon will be as the traditional proprietary data sources.
CAMERON NEYLON: We don't need to talk about the differences in coverage or completeness because we're getting to a point where those open data sources are just as good. I think really importantly, the benefits of scale and sharing that arise out of community and shared approaches to improving data make more sense in collective and community-based processes and organizations than they do in proprietary and closed systems. Services and analysis and intelligence and processing, these are all things that private organizations and proprietary approaches can work well on.
CAMERON NEYLON: But at scale of getting systems that give us data and flexible analysis for the kinds of things we actually care about, community approaches are not just better, they're actually going to be cheaper. So where I'm going to end with is really again coming back to this point. Do you and your teams want to be experts in reshaping questions, fit the available answers, or do you want to be the experts in helping researchers in institutions get answers to better questions?
CAMERON NEYLON: I'll finish there but just to say, if you're interested in more of the background of this, we have a book out. It is open access of course, via my M Press. And otherwise, I just need to thank a lot of people involved in this work. It is certainly not all work I've done. And the contribution the funding from a variety of sources. Most specifically, in fact, Curtin University which has supported this work very strongly.
CAMERON NEYLON: So with that, I will finish up and I will pass over to Mike.
MIKE TAYLOR: Thank you so much, Cameron. That was a really interesting talk. I really enjoyed both of the presentations so far. And I think given that we only had a relatively limited opportunity to communicate our ideas, I think that the three presentations together make a really nice matching set of talks. So I'm Mike Taylor, I'm a Digital Science. And I'm going to be talking about big data and computation, and what that means very much for the things that Cameron was just talking about, things that Rachel were talking about that increasing diversity and new ways of opening up and understanding the data that's available to us.
MIKE TAYLOR: And getting off that reliance on that one number to kill them all, one number to rule them all, not sure which one to go for. So I am going to start sharing at my screen hopefully. OK there we are. So hopefully you can see that. The presentation is called, what does big data and big computation mean for research impact and assessment?
MIKE TAYLOR: These are some absolutely enormous questions, which I'm going to try and deal with in about 10 minutes. This is me, I'm Mike Taylor. I've been at Digital Science for nearly six years, I was elsewhere for 20 years before that. I have a social science degree but I've always been involved in computation as you'll find up right from some very early days. I have just agreed a deadline for submitting a theses to my supervisor.
MIKE TAYLOR: So hopefully one day, my PhD will be complete. I expect Cameron and I have known about my research project since we both had much more hair and less of it was great. Oh, by the way, and if you want to grab hold of me, there is my email address digitalscience.com. I'm also on Twitter. So if you want to grab hold of me, tag me and ask me to talk about anything. I am one of those people who doesn't stop talking very easily.
MIKE TAYLOR: So I mentioned that I've been involved in computation for a very long time, not quite as long as the journal impact factor or using Garfield. Although when I started going to my secondary school in 1976, there was a computer science lab, it had a single terminal in it. It was dumb terminal, it was connected up to a mainframe somewhere, I don't know where. But in the middle of the computer lab, for some reason there was this cast iron device that could be used to punch holes in cards.
MIKE TAYLOR: It didn't look like this, it was much bigger than this. It was an absolute antique even in 1976. It wasn't being used by anybody. I guess it was being used by someone as a demonstrator model. It might be in a museum or would have been melted down. It was a massive device. But the thing that you're looking at is a card punch reader.
MIKE TAYLOR: And even now, when we people use Fortran in our computer language, it's still inherited some of the legacy from using columns to register pieces of data there. But I mentioned it because when we think about research assessment and calculation of metrics, the first time that anybody started calculating metrics like that was back in the 60s. Eugene Garfield, people of his generation back in the 60s, and it was taking months to calculate single numbers.
MIKE TAYLOR: And even then, they were using very, very simplified data sets, approaches which are still baked into the design of the general impact factor and other metrics. From now, we still are working with that legacy, which is decades old. But it took months to calculate those first data points. There used to be a blog that described it. Unfortunately, that blog seems to have disappeared.
MIKE TAYLOR: It's I can't find it anymore. A few years ago, 10 years ago, 12 years ago when I was working at Elsevier and we were working on the first iteration of the site space, sorry, the site score metric, we used to have to book time on not a mainframe set but a high performance computer. We would set up all the algorithms, we'd set up all the data. We would book time and set everything running on Friday evening and then go away for the weekend.
MIKE TAYLOR: Go out for beer and then wander home, come back into the office three or four days later. If our algorithms had failed, if they had failed, then we'd have to start again. And then I started Digital Science six years ago, and we were using a very early cloud-based thing. And I've put here it was a few seconds. Actually, it wasn't a few seconds. It was long enough making a cup of tea.
MIKE TAYLOR: So we've gone from decades in the 60s to 10, 12 years ago, taking a few days. A few years ago, it was taking long enough to make a cup of tea very important for this guy, long enough to wander off, make a cup of tea and come back again and see whether it worked or not. And now, I'm running those same calculations essentially free and in seconds. I had the bill for, my GDQ bill for December was, I think it was $0.42. It was an absurdly small amount of money that appeared on my credit card.
MIKE TAYLOR: So this is to give you an idea of what's happened to computational power and the ability to do those data. And I wanted to explore a little bit about what that's meant for how we can start pricing apart the data and getting some insights out of it. So as I say right at the very beginning and has been alluded to, we are coming out of a world where there was one number being used to rule them all, two if you include the H index.
MIKE TAYLOR: But these were relatively speaking straightforward calculations, that those general based metrics were very, very simplified to make them easy to calculate and to make it possible to calculate back in the 60s and 70s. And moving on, we've gone from having very simple number of data points to having many more data points even for doing things like the field weighted citation index, so article-based citation metrics or the field citation ratio, sorry, the relative citation ratio.
MIKE TAYLOR: Again, they're a little bit more complicated to calculate than a general one just in terms of the score, but they're still only based on that one single data source. There's no there's no diversity there. And as Cameron was just talking about, nowadays we have much more sophisticated needs, and as Rebecca was talking about, much more data that we can use to incorporate in our analysis. So the growth in computational power is really sparking our ability to do much faster, many more calculations, much more complexity.
MIKE TAYLOR: Coming back to the question that Cameron was raising about the answerability of the maths is a really interesting question. And I think that it's one of those things that certainly when I'm working with this maths, you have to think, how do I explain this? How do I put this in terms that it can be replicated? I'm not one of those people who believes in stuffing data into an environment and then crossing your fingers and hoping that whatever complex Bayesian calculations come back are actually an insight.
MIKE TAYLOR: You need to be able to track the data. And we as part of the academic world, we have that reliance both on research and openness and being able to interrogate each other. So just thinking about that shift from small data to big data. And as I mentioned, general metrics were very much simplified to reduce the number of data points to be able to put them onto those-- to be able to define them using a holes in cards and to compute them in Philadelphia in the 60s.
MIKE TAYLOR: So to calculate a general level citation metric really involves very, very few number of data points. Relatively speaking, this is quite an easy thing to calculate, quite cheap to calculate. And if we were going to do the same kind of approach, a very simple approach, then we're talking about perhaps between 40 and 40 million data points to calculate that for almetrics. And that's not even doing anything particularly interesting or complicated with the data.
MIKE TAYLOR: So for example, when I'm running the calculations that I was talking about in December, I'm adding in multiple levels of classification, many, many more data points to run it. So not thinking about 60,000 data points, not thinking about four million data points or even 40 million data points, but rather by the time we start taking all of these pieces of data into account over multiple years to try and calculate the dynamics of, we easily realize that we are calculating in excess of a billion data points over the course of this.
MIKE TAYLOR: And the reason we want to do this is because we want to answer some of those questions that the Cameron's raising. Some of those more complicated questions about dynamics and how a policy in publication funding for example, how is that affecting the downstream impact of your research? Because of course, as Rebecca has said, impact is not a single thing. We're not talking about simple things like citations, rather than we're thinking about public engagement, we're thinking about economic impact, we're thinking more about diversity, who is citing us?
MIKE TAYLOR: Who's reading us? Who is engaging in our research? And do that. We now have these platforms, we have access to the data, we have access to the computational power, and it literally takes seconds. I cannot make a cup of tea when I'm running this calculation. This calculation that exceeds over a billion data points, when I calculate it, it is taking seconds.
MIKE TAYLOR: I have a barely enough time to take a swig of water, not a time to make a cup of tea or go on holiday as it might have been in the 60s or 70s. So this computational power, the access to the data. The fact that so much of this data is open and available to us to compute, it means that we can start answering much more complicated questions. But to do this requires something of a shift in mindset.
MIKE TAYLOR: Because we seem to have evolved out of this world where we had this one single number, one set of numbers to answer questions. But we started trying to answer many, many questions using the same numbers. And this was a consequence of, what was the only number we have. We only had these single numbers. So we were trying to use the same tool to answer a number of different sets.
MIKE TAYLOR: This world that we're living in is much more complicated. And I think that we've seen that there's quite a lot of resistance to change. Some of those approaches that we've been talking about that go back years and decades have been very robust, they've been criticized, quite rightly criticized, but there's a reluctance to change in the world. As I'll mention it, it's very variable whereabouts that reluctance, whereabouts to that reluctance is firm.
MIKE TAYLOR: But questions like, what does excellent research mean? They are really hard-- they are really hard to answer partly because we try to express questions that could be answered in the data that we used to have to use. And the world of big data and big computation, faster computation, means that we really need to step back from the problem and start really addressing those really addressing ourselves to the questions that we are able to answer.
MIKE TAYLOR: So rather than asking quite primitive questions, if you like, of how much excellent research, we can start defining what excellent means in terms of the data that can answer those questions. So that primitive question, how much excellent research can you publish? We can start asking much more complicated questions like, how much excellent-- in terms of that, what excellent research means.
MIKE TAYLOR: Does it mean more diverse? Does it mean we want to be cited by the Lancet in the New England Journal of Medicine? Does it mean that we want to have a broad, a diverse? I've got a really, really nice example of one of those. So I did some work for an academic institution in the US. And they were using what I would say without mentioning who they are, and I'm not going to mention who they are because they've sort of moved on.
MIKE TAYLOR: But two or three years ago, the question that they were asking themselves to understand how much excellent research was very much based on whether or not their academics were publishing in the New England Journal of Medicine, the Lancet. It was fairly, that's fairly simple because they had a certain threshold using a journal metric and excellent was a line that they sort of arbitrarily drew around that point.
MIKE TAYLOR: Now, when I came in, we were having a look at some of their data and trying to understand what was excellent. And to Cameron's point about diversity, we were able to identify one of their researchers who had never gone anywhere near the New England Journal of Medicine or the Lancet, and was publishing in open journals. I think it was Post-Tropical Medicine.
MIKE TAYLOR: But her research was having an enormous impact globally. And you were looking at the diversity of the citations, looking at the diversity of health metrics. And from my perspective, my argument was, these are the people you should be celebrating because the impact they're having is measurably much more than someone whose article may or may not be getting any citations but was in the New England Journal of Medicine.
MIKE TAYLOR: So the new world, the old world, very arbitrary thresholds. The new world, much more complicated, much more sophisticated. And really, we have the ability to answer questions which are much more precisely defined, much more attuned to the data that we have, and it's much faster for us to explore these answers. We don't have to make these big investments and huge clunky systems, rather we can be agile, we can be interrogative, we can form hypotheses and we can test those hypotheses using data that anyone can run.
MIKE TAYLOR: I did AI did a very quick little example because I thought this was quite fun. So I took a set of three different journals that look as if they're the same sort of journal and plotted out some data. So I ran this data just for this presentation. So back in the old days, we would have had one citation-based metric to compare these three journals.
MIKE TAYLOR: Now, we've got a world of data that we can compute to understand much more about the performance of an individual journal. So these are-- sorry, I left a line in there from a previous slide. So these are three journals, International Dairy Journal, Journal of Dairy Science, Journal of Dairy Research. So they look very similar to each other on the surface. They sort of publish science from more or less the same areas.
MIKE TAYLOR: International Dairy Journal appears to be slightly more technical, slightly less veterinary. JDS publishes about 900 articles a year. Journal of Dairy Research is a little bit smaller but nevertheless, they're more or less all in the same space. And I thought it'd be fun to have a look to see what big data can reveal. So really nice example here is sort of patterns where we can clearly see that the International Dairy Journal gets far more patent activity around it.
MIKE TAYLOR: Interestingly enough, if you start breaking out how well it's agricultural science papers perform compared to the other, you start getting some different things. But this is data that's being driven from volumetric data, it's being driven from dimensions, it's being normalized by subject areas on a very fractional basis. And we can see that there's a distinct difference between those two journals.
MIKE TAYLOR: If we have a look at the news. So the relatively the relative standing of the news coverage. So in this case, the Journal of Dairy Science was this medium lower journal for getting news coverage between 2010, 2014. Something seems to happen in 2015 because subsequent from 2015, that journal gets a lot more news attention than either of the two journals. I get looking very, very similar.
MIKE TAYLOR: This is computed over all of those data points, millions of data points. Tweets, really, really interesting. And again, this one journal, Journal of Dairy Science seems to have gone, seems to have been turbocharged in 2014, an enormous uptick in the amount of Twitter coverage from it. And again, these aren't counts, these are normalized balanced by all of the different subject areas.
MIKE TAYLOR: And I could break this down by diversity, I could break this down by open-access status, I could look at it in all different ways. The one thing I can absolutely say, this has got nothing to do with the rate of citations. So that ordinary metric, if we have a look at the ordinary metric. In this case, we can see that the Journal of Dairy Science, the one that's been showing that enormous uptick in news coverage and Twitter coverage, nothing to do with the academic standing of it.
MIKE TAYLOR: The citations for each one of those journals apart from the last year is relatively static. So we've gone through several stages over the course of the decades. We've gone from having legacy metrics, metrics that were based on old ways of doing computer closed data, very, very hard, very expensive. Lots of investment had to go in to calculate those numbers. And we've moving away from that to a world where we've got open data, where we've got fast data, where we've got big data, where we can do all of this hypothesis testing using data science approaches, where we can shape the questions and come up with more agile answers.
MIKE TAYLOR: Now, my question I suppose that I've left thinking is, well, is the academic research evaluation process lagging behind the data, behind the data science? I think it is. I do a lot of work with organizations that do research from all around the world and in many, many different sectors. And it suggests to me that the corporates are much more willing to accept a data science approach, this data approach. They're much more interested in agile approaches to ask questions about research and answer questions about it.
MIKE TAYLOR: Whereas, absolutely in academia, we do need to be a bit more cautious, a little bit more careful about how we develop. I'm still looking forward to the next five, 10 years. I think is going to be really exciting for research assessment. And with that, I'm going to stop and hand back to Melroy.
SPEAKER: Thank you very much for that Rachel, Mike, and Cameron. [MUSIC PLAYING]
Form Title Download
Video
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode