Cadmore media player playing video The discovery audit revisited-NISO Plus 2022
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
The discovery audit revisited-NISO Plus 2022
Description:
The discovery audit revisited-NISO Plus 2022
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/8ad871d6-0695-4361-8aa3-69b0c6b38fc9/thumbnails/8ad871d6-0695-4361-8aa3-69b0c6b38fc9.png?sv=2019-02-02&sr=c&sig=aSI36fq3ZPLVZ5yre46qm5xZxhGk%2FDVz9k0ebCL7p98%3D&st=2024-12-22T02%3A04%3A48Z&se=2024-12-22T06%3A09%3A48Z&sp=r
Duration:
T00H59M05S
Embed URL:
https://stream.cadmore.media/player/8ad871d6-0695-4361-8aa3-69b0c6b38fc9
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/8ad871d6-0695-4361-8aa3-69b0c6b38fc9/The discovery audit revisited-NISO Plus 2022.mp4?sv=2019-02-02&sr=c&sig=G6i0OmWB8chI%2FmMC2WtIjCiYHK%2F1XF%2B2pdgRO822wf4%3D&st=2024-12-22T02%3A04%3A49Z&se=2024-12-22T04%3A09%3A49Z&sp=r
Upload Date:
2022-08-26T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
PETER: Hey, welcome. As the program description outlines, this session seeks to examine how audits of many unseen factors which can impact the use of library discovery tools can help identify disconnects and incorrect information and how solutions can hopefully be forged based on what is found. We all know that publishers and libraries both want licensed content to get used.
PETER: So it's more important than ever as reliance on discovery tools increases to make sure that all is working as intended. And seeing how the factors ranging from metadata to holdings lists to links and authentication, all have to be checked. And audits of all these components can help identify what may need attention as the speakers you will hear from will outline. Today we have presenters with perspectives from both libraries, those being Northwestern University and the State Library of Ohio.
PETER: And publisher content providers, those being the IEEE and Springer Nature Publishing. And we hope you'll find much value in their experiences and their findings and their recommendations. And we'd like to start off the program with Geoff Morse from Northwestern University.
GEOFF MORSE: Thank you, Peter. Welcome. My name is Geoff Morse and I'm going to talk to you for a few minutes about the experience the Northwestern libraries had in filling out an Open Discovery conformance statement. So here's my picture and my current position.
GEOFF MORSE: I've been with the Open Discovery Initiative from NISO or ODI since 2018. So the NISO Open Discovery Initiative, to provide us with a little introduction, began in 2011 at the American Library Association's annual conference in New Orleans. At this time there were several newly emergent discovery tools that provided large aggregated index of content. However, there wasn't a lot of transparency at this point about how the content was provided, what was indexed or how conforming all of this information and material was with the standard expectations of libraries and librarians.
GEOFF MORSE: Keeping in mind that up to this point, libraries and librarians had been pretty comfortable and knowledgeable with the OPEC and knowing everything that their user could potentially search. So the Open Discovery committee was formed and made up of representatives from content providers, discovery providers and libraries to address some of these concerns. There are several overarching goals of the Open Discovery Initiative.
GEOFF MORSE: And first, really it is crucial that ways are defined for libraries to assess discovery systems and the content available within discovery systems. How libraries-- Excuse me. How librarians and users know what is included in a discovery system is essential really to anyone doing research. Secondly, it's important that processes are streamlined for content providers so they can easily work with discovery service funders to include their content in a discovery system.
GEOFF MORSE: There have also been concerns about fair linking as discovery service vendors at times are also content providers. And to address this concern around fair linking, the Open Discovery Initiative has to find models for fair linking from discovery services to the publishers of content. In the fourth overarching goal of the NISO Open Discovery Initiative or ODI is to determine what uses to systems should be collected for libraries and the content providers so we can better assess how these new tools are being used.
GEOFF MORSE: So I'll just provide a little bit of history about Northwestern's experience with discovery systems. As you can see, we've been using a discovery system Ex Lib versus Primo for a number of years. We initially started implementation back in 2008 before there was a centralized discovery index. And during our implementation, we are already thinking about adding additional records from some of our various digital collections to the discovery system.
GEOFF MORSE: And creating workflows, transforming these records from EAD and MODS into XML format. And Additionally during this period, the discovery system or since that period, the discovery system has evolved really from a secondary system kind of backing up our OPAC that was being aimed really at beginning researchers and undergraduates to really the main search system encompassing the local catalog at Northwestern as well as digital collections and the large central discovery indexed managed by Ex Libris.
GEOFF MORSE: So the NISO recommended practice document that the Open Discovery Initiative is created includes general measures as well as measures related to various aspects of configuration, communication, documentation and training. There's a checklist for discovery service providers to meet the recommended practice for Open Discovery. There's a checklist for content providers, and there's a checklist for libraries which is what I'm going to be referring to today mostly.
GEOFF MORSE: The checklist for libraries focuses on the responsibilities of libraries to configure their discovery systems to a level of optimal performance. Areas such as staff roles documentation and collaboration are covered by the checklist, which serves really as a guide to assist libraries to get the most out of their discovery system in line with the goals of the Open Discovery Initiative. In some ways, the process of doing the conformance statement checklist which was undertaken starting in May 2021 and finishing in late November, in some ways the process of doing the conformance statement has improved the mutual understanding at Northwestern amongst library staff of what others are working on.
GEOFF MORSE: It also is facilitated communication and helped place in context the work individuals are doing on the discovery system. In 2021, the library's Primo team, the team charged with overseeing the operation of our discovery system, started as I mentioned in May, started to fill in a library conformance statement or checklist which we saw in the previous screen. This was an opportune time to do this as the Primo team had recently been reconstituted with a number of new staff members.
GEOFF MORSE: So the first bullet point there relating to understanding of workflow and staff rules. The understanding of staff assignments related to the discovery system was clearly very important. While no work assignments were changed as a result of working on the conformance checklist, importantly it did become more clear who is taking responsibility for certain areas of discovery configuration.
GEOFF MORSE: The advantage of knowing who is responsible for specific roles is very helpful, particularly in the resolution of issues or problems. Cross-team communication improved as a result of this work. It's also worth noting that particularly in large organizations, and Northwestern libraries are fairly large, tracking down the right location for a role, whether the work is already being done and not documented or work that should be being done but isn't can be a challenge.
GEOFF MORSE: Large distributed organizations like ours with discovery related work happening across multiple domains and areas, often results in not everyone knowing what everyone else is doing so. And I think the conformance statement really helped us in this way as we got a better understanding of the big picture in the context of where the discovery system lay.
GEOFF MORSE: The process of filling out one of these statements required interested parties to meet and work together to establish roles and understand the larger context of the discovery system within the library. As an example, the Primo team is much more aware of the work that the electronic resources librarians do in terms of selecting databases and collections for activation and the central discovery index.
GEOFF MORSE: Library staff are better informed on potential issues with this work as well as whom to contact with very specific problems. Library staff now regularly meets monthly to discuss configuration options and new developments in the discovery system. This may seem straightforward and a pretty standard practice.
GEOFF MORSE: But it's been very helpful to regularly convene the group of people from across the libraries to work on this. These meetings enable open discussion of configuration options. To answer questions about the activation of the correct database collections, the appropriate people in the electronic resources area mentioned in the previous slide of the library had to be consulted.
GEOFF MORSE: Upon consultation, the electronic resources people could share some of the complexities behind knowing which discovery package to activate, which many of us weren't aware of just how much and how hard they had to work on that. This is facilitated better communication among members of the team and the electronic resources staff. And has enabled all of us to get a better understanding of the big picture in which the discovery system operates.
GEOFF MORSE: The Primo team is more familiar than ever now with the official documentation and will contact the vendor with questions and problems. Library staff are now more aware of whom to report problems to with the discovery system. And we've also facilitated with new online forms an easy way for staff to report any issues they see as soon as they see them in the discovery system. It results in faster resolution of the problem.
GEOFF MORSE: So again, the discovery system checklist brought a lot of these issues to the fore when we started discussing them and really helped us improve our approval processes. The conformance checklist also emphasized the importance of participating in the larger community of discovery system users to both gain information and receive assistance, as well as provide assistance to the larger community.
GEOFF MORSE: Several members belong to the Primo listserv and the library uses and contributes to an idea exchange for Primo users, where new features are suggested and acted upon sometimes by the company. And staff members also contribute to and monitor the developer's network site as well, where you can put up new features you've custom made and allow other institutions to use them or vise versa use new features that have been created by other institutions.
GEOFF MORSE: So again, the conformance checklist helps bring those issues out and emphasize them. So really the process of filling in a conformance statement has helped to highlight our need for improved documentation. Well, now without documentation of decisions that we have made, quite often that documentation has been dispersed in various places and may be lost if a staff member moves on to a new job at another institution or even another department or work area.
GEOFF MORSE: So this is one area it became clear we really have to improve. Another area for improvement was that of staff training, whether there have been sessions for staff when the discovery system was introduced or an occasional presentation on new features that really has not been a systematic training of library staff on the discovery system. And since it's our front and center discovery tool now, it seems important that we work on improving that in the coming year, which is one of our goals.
GEOFF MORSE: So just very quickly in closing, the discovery checklist from NISO Open Discovery Initiative has really helped us focus some of our discovery work and really made our discovery system work better with better communication and hopefully more optimal use for our user community. But it has also made it easier for us to work with content providers, to work with our discovery provider as well.
GEOFF MORSE: So thank you and I'm definitely open for any questions.
PETER: Thank you, Geoff. And our next speaker is David Greene from the State Library of Ohio.
DAVID GREEN: Hello, and thank you, especially thank you to the group for inviting me today. I am David Green and I am from the State Library of Ohio here in Ohio, Columbus, Ohio. And before coming to the state library, I was actually working for a non-profit consortium called OhioNet, which is a non-profit organization that supports all types of libraries across the state of Ohio be they academic or public or special.
DAVID GREEN: Whatever the case might be. And it was actually at OhioNet that I got the bulk of my experience doing what we're calling the discovery audit where I did not just one or two, but up to a dozen audits for various libraries with the state library being probably my most recent one that I have done. So how did it come to be that I managed to do so many discovery audits you might wonder.
DAVID GREEN: And a lot of it has to do with the relationship of many of the libraries to our statewide consortium called OhioLINK in the state, which has to be honest, I'm not even sure on the number at this point, but maybe 100 members academic libraries across the entire state. And it was probably, I think in 2013 that OhioLINK had negotiated a deal for many of its members to opt in to a discovery layer with EBSCO, with EBSCO's EDS at that point.
DAVID GREEN: And as you can imagine, the rollout was going to require a lot of effort. And that's where the audit came in post implementation for a number of those. So now that you know a little bit about my background, we can talk a little bit about what it looks like an audit for one of these libraries. And in every single case with every single library, these always began with discussions with library staff to come up with what we felt was the most authoritative list as possible of what our entitlements were.
DAVID GREEN: And as you can imagine, this was always easier said than done. As I mentioned, my first libraries were not just in a consortium called OhioLINK but they were actually a consortium within a consortium. So the libraries I tended to work with in the beginning were part of a consortium called the Ohio Private Academic Libraries or OPAL. And right away you can begin to see how this is going to add layers of complications when it comes to figuring out what do we own, what are our subscriptions and who do we get it from.
DAVID GREEN: As you can imagine like a library may have its local subscriptions that they purchase directly. They may have subscriptions through OhioNet. They may have subscriptions through OhioLINK. And then to make things even more complicated we also have some statewide content that's available to libraries to with statewide agreements. So picking it piece by piece, we had to figure it out who do we get it from, where do we buy it.
DAVID GREEN: And of course, as you begin to work through the and identify where these things are and what they are and you'll hear me maybe repeat this again and again but document, document, document. As a sort of extra layer to figuring out where something may have come from, one of the things that we realize right away too is that while it's easy to identify what may be current subscriptions are, we find that there's also a wealth of content that may have been perpetual access that we have purchased in the past.
DAVID GREEN: This is especially true when figuring out things that have been purchased with perpetual access at a consortium level, 1 or 2 levels up. So having a historical record of some of that stuff was incredibly helpful for identifying this content. And again, once you find these things do not just state the obvious but document, document, document. Once that preplanning is out of the way, I think that the audit itself came down to pretty much three major themes that you see here, content, access and user experience.
DAVID GREEN: This is how I see it anyway. And in terms of content, we're just talking about identifying and enabling the content itself, whether that be anything from metadata to full text. The content that you intend to be discoverable should actually be discoverable. As far as access goes, issues of whether or not you can actually get to the content. Be it navigating to it through the platform itself or navigating to a place where you can.
DAVID GREEN: And then issues of user experience as well is the content and the access configured in a way that makes sense to the user. Is it easy for the user to Find does the look and feel look good for the user? This also may sound like it's sort of a linear sort of process in terms of the-- I'll add that it's not at all linear and it is, in fact, a rather reiterated process.
DAVID GREEN: So if you haven't done one before, you can expect it to be, I don't want to say messy, we'll say reiterate. So in terms of content, we relied heavily on some of that preplanning work where we take our lists, usually broken down by the content provider and ensure that the content is configured in the discovery layer to begin with.
DAVID GREEN: And depending on the discovery layer, this might take a few shapes. So in making sure that the source of that content in EBSCO case, they call them partner databases, were enabled. At the same time discovering things that needed to be disabled or take them back out that were no longer relevant. Also finding knowledge bases and packages to configure. And while doing that, trying to discover when is the last time a knowledge base has been updated.
DAVID GREEN: If it hasn't been updated in a while, maybe that's a clue to get in touch with the vendor to talk about that. And also at the same time discovering that knowledge base packages their names frequently shift or change. And are we actually identifying the correct content? Whereas it may be something that looks similar but is actually something else.
DAVID GREEN: Choosing the right one we learned is often very difficult. And then there were some situations too where we wanted to configure content that maybe we didn't necessarily purchase or have access to but for purposes of in inter-library loan, for example. So we had not frequently, but some libraries that wanted to identify certain pieces of content make them show up in the search results.
DAVID GREEN: But only because it was highly relevant to their audience or to their students, and they wanted to at least give them the opportunity to be able to make inter-library loan request for that kind of content too. As far as access goes, we're talking about getting to the content or again, being given an opportunity to find the content on another platform.
DAVID GREEN: And as far as linking goes, I'm primarily referring to the links that are inherent to the Discovery platform. And so rather than say open resolvers. Not to say that we didn't look at open resolvers because we did and they are important. But in this case, the links that we needed to reveal were EBSCO specific. And one of the things that is not always obvious to staff and to librarians is that it's not simply enough to identify and enable the links where appropriate.
DAVID GREEN: But that there's some underlying configuration that still needs to happen. Usually this takes the shape of required parameters, for example. Usually say a customer ID, sometimes a package ID that needs to be added to the linking scheme, and without it access fails. That wasn't always obvious at first and we had found by slowly taking the process of the audit that these things were going unnoticed.
DAVID GREEN: Similarly, I found that inter-library loan links were also something that were not necessarily available by default or were never configured. Discovery platforms will usually allow you to configure either something that's baked into the platform or at least a third party ILO option. The baked-in option is what we generally did. But for all links, be they ILO or otherwise, it took some time to configure out how or figure out how to actually configure those.
DAVID GREEN: So if they were truly available for the content that we were trying to link people to. As part of that by the way, it also took a little bit of time to learn how to suppress certain links under certain circumstances so as to not overwhelm the end user. So for example, we didn't want inter-library loan links displaying on all content, only for content that wasn't available.
DAVID GREEN: Or you don't want to overwhelm your user with say 20 different options on a single record when maybe only two or three of those are the most relevant or even preferred links for your user. And you can expect that will take some time to figure out as well. And then finally with authentication there being towards the end, one of the first things that we worked through was to back up for a moment and review like IP addresses, where the IP address is still what they needed to be a configured at the platform level.
DAVID GREEN: And this often required maybe some help with some IT staff. It's one of those examples where it helps to have a working relationship with an IT department because they don't always know or think to tell library staff when there are additional IP addresses to add or that maybe sometimes they change. Similarly, all the libraries that I have worked with use EZproxy for their off campus authentication.
DAVID GREEN: So part of the audit process also required us to ensure the EZproxy was configured properly and that all platforms that were represented in the content were configured with EZproxy likewise. And then finally, something that, again that surprised us was how often people were denied access to open access content. And that was frequently because the platforms on the other end were not things that were configured with EZproxy.
DAVID GREEN: And the solution there of course, was to go back and identify what is our open access content that we have enabled. And then to turn off the proxy for those so that it wasn't blocking access for the user to get to that content. Again, something that we may not have thought about initially during the implementation phase but discovered more through reporting of users that they were not able to get that content and found a pattern there.
DAVID GREEN: So finally with user experience as you can imagine, once the agreement with OhioLINK came out, the implementation was maybe a bit rushed just given the sheer number of libraries that needed to come on board. So a lot of it came back to just the basics. Do the primary accent colors reflect my institution's brand and identity?
DAVID GREEN: Are there logos being used that reflect my institution's brand and identity? Are icons configurable? Can we change them to be more relevant to the users or to match the overall brand and identity? Are there any broken images that need to be fixed? How are they being linked to? That sort of thing. In addition to that, it also helped to review just basic information, such as contact details, hours or even simple things like navigating back to the library's main website.
DAVID GREEN: As an extension of that user experience, one of the things we also did during an audit was to review what language was being used in the discovery layer. And by this, I mean, how do we refer to things or what do we name them? What do we label them? Does the language that's being used by the interface say with limiters or otherwise actually make sense to your users?
DAVID GREEN: So for example, if your primary audience were undergraduates, a number of them might not understand what inter-library loan means. So can we change the language of that link to be something actionable instead, such as request this item or find this item, whatever the case might be. Generally, you would have flexibility in how things are labeled in a discovery layer.
DAVID GREEN: So it was just a process of trying to review what our options were and then thinking critically through like improvements to those where we thought they were necessary. And finally, some takeaways. And I'll mention here that back when the discovery layers were first being adopted, we certainly didn't know about things like ODI or conformance statements.
DAVID GREEN: We didn't necessarily have those at our disposal. And so things are definitely moving in a positive direction at this point where we do have that sort of structure in place now to help us with the self audit, especially now that I'm at the state library. These are things are becoming very useful. And we also didn't have things to such as documentation provided by vendors themselves that sometimes you will find now like say health checks or system reviews.
DAVID GREEN: And those things, I think, are finally coming out of these conversations that we're having with content providers. So important takeaways I think of this is one of the things that wasn't so obvious for us to do when we first started but is maybe something that I would insist upon now is that before we even begin an audit, you want to record a baseline so that you can see how changes may improve your discoverability and the usage of your discovery layer.
DAVID GREEN: Just as an example for that, at the state library, we had not yet set up Google Scholar and implemented that with our discovery layer. And once we did, it was only one month later that we found that 29% of the traffic that was coming to our discovery layer was now coming from Google Scholar and that was without even advertising it or even mentioning it.
DAVID GREEN: We simply just set it up and people found it. And that was a very impressive and surprising. Something else you might want to consider as you're doing this is to create a separate profile or environment if it's available. Not only will this allow you to do your work without interrupting people who are currently using it, you can sort of do your work siloed. But it also gives you a opportunity to slowly troubleshoot and configure things in a way that's maybe a little more structured.
DAVID GREEN: And then even once you do go live, becomes a nice backup that you can refer to as well. Also talk to content providers. Like I said in the beginning, we didn't have things like conformance statements to refer to or even to begin some of these conversations. But thankfully, those things exist now. But in addition to things like that, also ask your content providers how you can stay in the know for changes in content.
DAVID GREEN: So for example, do they have content alerts? Do they notify you've changed to packages to package names? If they provide those sorts of things, it's helpful to be aware of them so that you can take action when something does change. And then finally, I would also suggest that reviewing the collection development policy at your library. I think because Discovery was so new to everybody that the approach tended to be an all lean approach.
DAVID GREEN: And it was one of those things where we didn't stop to think should we, it just could we. But that may not have been the best approach in retrospect. So I think maybe a better approach moving forward in terms of doing a self audit is to consider what the collection development policy is. The content, just because we have the content, does it necessarily fit our audience and what we're trying to provide?
DAVID GREEN: Should I have limited content for example, that maybe wasn't relevant to my users? So I think that might have been something that we probably would have done differently had we considered it at the time. And with that in mind, I am happy to answer any questions about doing any number of these audits or just to talk generally about them. And once again, thank you very much.
PETER: Thank you, David. And I think everybody is seeing a real persistent theme coming through the presentations as it relates to all the different factors, any one of which can get in the way of smooth operations. So lots for us all to think about. And now to look at things from the content provider perspective, the publisher perspective. Will start off with Robert Lacey from the Springer Nature Group. Bob?
BOB BOISSY: Yes. Thank you very much, Peter. And hopefully, I can build a little bit on what you've heard so far. This is Bob Boissy speaking to you. I am Director of Account Development at Springer Nature, in the US and Canada. My relationship to the ODI Group is liaison to the ODI Working Group, from NISO Information Discovery and [INAUDIBLE].. And I know that the NISO folks would like for all of the people watching today to be thinking about what might be other work that could even enhance what we're talking about with discovery today.
BOB BOISSY: What things could possibly be built [INAUDIBLE].. So with that, I will just give you my information provider, publisher perspective on the context for discovery audits, and how we might be re-visit it. The one thing to know, coming from my point of view-- I do work for a large publishing house. And like large universities, we're lucky enough to be able to have some specialists.
BOB BOISSY: So we have folks in account development and libraries that are really specialist library support people. That means technical and promotional support. When we monitor usage-- and monitor usage is just common sense. We want to know if there's a problem before it goes too far. And so we do monitor the usage of our content with our clients.
BOB BOISSY: And the idea is that our librarians have told us if things have been going well and then they start to not go well, their first inclination is to say, that is probably a technical thing. And so we are trained to go looking for technical things first. And if it is not a technical thing, that we find no problems with the pipeline, then you might switch over to talk about some disability or other promotional type work [INAUDIBLE] on campuses.
BOB BOISSY: But the idea of a discovery audit is that we take a proactive approach to what is in fact an uncertain situation. We don't really know ahead of time how people will take up our content. So this discovery audit is to make sure and rule out technical problems. The basic elements that have been around since the mid 2000s, the publisher discovery audit, are basically that we know that we can start by looking at a library's website.
BOB BOISSY: We know that in the United States and Canada, and many other countries as well, libraries open up their catalogs, their discovery tools, their search boxes to the public. And that allows us to do a discovery audit. I have, as a matter of fact, done discovery audits with corporations as well. But that is [INAUDIBLE]. They don't open up their discovery tools to outsiders.
BOB BOISSY: Usually, I'll do that directly with them via some kind of video reading, where I ask them to search for certain things, and they come back with certain results. And we check on discovery kind of the strength of the discovery system together, sitting at our screens. So the main thing that you do during discovery audit is we will look for our own content on your systems.
BOB BOISSY: And we will do increasingly specific and use unique text elements from things like titles and abstracts and full text of content, to see if our content rises up in the results. But that's basic discovery audit. We'll do a sampling. We'll see how well things are found, and if our content is high enough in the search results that we think that makes sense for what we have searched.
BOB BOISSY: We do check Google Scholar Library Links, but a lot of libraries have that up now. We think that is an element that's important these days. We will also just be sure that you understand that we're checking book and book-chapter information as often as we check journal articles. So we do kind of an all-over-the-map type looking at the content. But today's talk is about going beyond that and revisiting the discovery of it, and perhaps amending it.
BOB BOISSY: And I've been working on this now for a while, since I joined the ODI request liaison, and encouraged library conformance statements. Perhaps out of self-interest as a publisher, I wanted these ODI statements as a starting point for a conversation. What I've realized is that it really helps to enlist the libraries earlier in discover process-- not to do it on my own, but, in fact, to work with libraries on it.
BOB BOISSY: Let them know that we're taking a quick look, that we're checking up on things. And hopefully, we're doing it together for the same reason, that we want the content to be well used. So I will check for an ODI statement. But if there is no ODI statement prepared, I've found now that I can use the checklist as a discussion, that I like to share-- so please share the checklist-- to show why I'm asking certain questions to the library.
BOB BOISSY: And I can focus on things, perhaps, that will [INAUDIBLE] to make things better. One item that I will point out is a truism, I think, that a picture is often worth a lot of words. So I have taken it up to really make sure that captions for graphics are discoverable. Many researchers are on the hunt for a graphic that they can re-use.
BOB BOISSY: And so good captioning for graphics is very, very important. And I want to make sure that our graphics are discoverable. I will also mention that KBART automation is relatively recent. And we do check for this. If it's not in place at a library, we might talk about that-- KBART automation being a way to get your holdings updated regularly with your discovery service, without you having to do that extra work manually.
BOB BOISSY: But once it's set up, it works. So in the ODI process, other NISO standards come into play-- KBART automation and KBART basic formats is part of that. I don't know if this is my personal background, or whatever, but I think it's not a bad idea to confess a little bit. It's OK to have imperfections in your discovery environment. That is expected almost.
BOB BOISSY: I will confess certain things that maybe aren't as good as I would like them to be in our metadata. With a client, might as well lay the cards on the table. But I think our metadata is largely good. And I think most publishers really work hard at it. But maybe there's imperfections [INAUDIBLE] side, binary side, or the intermediate or even a discovery layer could be some kind of work.
BOB BOISSY: And the only way to really go through that is to talk about it together. So for this purpose of this presentation, I did have a back and forth with Jeff at Northwestern about his checklist, and how they filled it out-- two pages from me and two pages back from Jeff. And that exercise, this was really very helpful [INAUDIBLE]. I sent the other folks who are presenting today a copy of that.
BOB BOISSY: So they know that we really did have quite a good conversation over the [INAUDIBLE] statement. And I was interested in the system default settings. Are they readily apparent? Are you confident that you can change them? Can you ask for changes? And the documentation of configuration decisions is very good. Whenever there's documentation, I'd love to know how public that could be-- and how open we can be and how level a playing field we can make for ourselves when it comes to sharing in relation to good discovery.
BOB BOISSY: And I like to point out to people that even though we're very much kind of a high stakeholder when it comes to discover-- in terms of if discovery doesn't work well, our mutual content does not get used. And that's our downfall. So we really are invested in it. But most of us on the publisher and aggregator and information-provider side have never actually operated in the [? Discovery ?] system, or a catalog, or a knowledge base, or a URL in a link resolver.
BOB BOISSY: So we rely on you to help us with terminology, with telling us how things work. And the library performance statement is a great step in that direction. Yeah, I am also interested in little bits and pieces of marketing. If some information resource really has a nice feature to it, could that possibly make it through the metadata and on to the discovery layer, so that that resource could show up is particularly good.
BOB BOISSY: If it weren't, if the feature was excellent, sometimes people think about, does this database [INAUDIBLE] patent data. Patent data is an extra [INAUDIBLE] across the board. And I'm just kind of throwing that out there, letting people know up front that it's some kind of feature that's available to support. Jeff and I talked about the selection of packages.
BOB BOISSY: And that's still being something of a problem. And so I [INAUDIBLE] myself package ID work, which is just starting up now. And I also want to let people know that you may talk about metadata, and you may think metadata is only [INAUDIBLE],, but not necessarily. Some metadata may be in there because it serves an electronic resource management role-- that is, if they have a role in assessment of resources.
BOB BOISSY: And if that data is needed, then we have to talk about that. Also, I just like to point out that, yes, we would like to be part of the community that talks about [INAUDIBLE] with you-- maybe with your discovery service providers-- we allowed on some generalist listservs. And we do monitor them, but we're not allowed on others, which possibly is something to look at.
BOB BOISSY: The last slide here for me is just to point out that we do have common ground. We both want licensed content to be used. We understand [INAUDIBLE] the discovery that leads to use. We want to together get the discussion going about improving metadata. It could be the handling of metadata-- the use of it-- and improving mutual support. Where can we get the [INAUDIBLE] better, to answer some of these questions that come up [INAUDIBLE] they are?
BOB BOISSY: And how can we be more transparent about the decisions we make as a provider, and you make, when it comes to discovery? And of course, my personal mission is to remind people that one good reason to keep supporting [INAUDIBLE] library [INAUDIBLE] library engineered based discovery because library referrals to resources almost always result in full text success for the end user, while public search engines deliver up the vast majority of denials.
BOB BOISSY: And I've done studies to prove this. And that means some unsuccessful outcomes [INAUDIBLE].. Even though some of those starting may be popular, they can result in bad things. So let's keep after library discovery. And that is it for me. So I will stop sharing. And let us slide over to Julie.
PETER: OK. Thank you, Bob. And as my mind is reflecting on so many key themes that have come up-- David talking about everything from using terms that our users understand to the names of packages changing. And now you've introduced this whole theme of content providers needing to get on these systems and see how their own content is working within them. That's all part of how we manage this ecosystem together.
PETER: So to tie so many of these themes together, Julie Zhu, from IEEE, who has a formidable role there in watching over their content in all the different types of discovery platforms, is going to give us a view from her perspective. Julie?
JULIE ZHU: Hi, I'm Julie Zhu, IEEE Senior Manager for Discovery Partners. I've been on the ODI Standing Committee since 2014. I will review the overall major areas for discovery auditing and point some new directions. As we all know, library discovery systems include complex data flows. And here's a flowchart of data flowing from publishers to vendors, discovery indexes, and knowledge basis, and then to libraries.
JULIE ZHU: Data flows to CrossRef and authentication systems, like it's EZproxy and [INAUDIBLE],, also have major impacts on the item's discovery, linking, and access. In 2015, I did an auditing study to examine the availability of library-subscribed content in 24 for library discovery systems.
JULIE ZHU: These 24 libraries are from different parts of the world, using four different sets of discovery service and link resolver tools. But they all subscribe to the same content. The results vary a great deal among the libraries. I found significant content gaps in the publisher systems and workflows and the vendor tools. And also, over half of the library had real problems in their discovery system and the link-resolver-tool configuration.
JULIE ZHU: So the causes for the content gaps are many. The problems can come from the publisher site, starting from the incorrect author names, or insufficient keywords, or other metadata in the author submission systems, all the way to publisher-content delivery to vendors, discovery indexes, and knowledge bases. The problems can come from the vendor side, in indexing, linking, displaying, and more.
JULIE ZHU: A considerable portion of the issues actually come from the library side. And like I mentioned, over 50% of the libraries may have issues in selecting the correct collections in the discovery index, the KB tools, and enabling remote access in various authentication systems' tools.
JULIE ZHU: So what can publishers do? They can start with the quality and delivery of their own content. The ODI content provider conformance statement is a very good start. [INAUDIBLE] was among the first publishers to publish the content provider conformance statement, in 2015, and then in 2021. In my 2018 article "Should Publishers Invest in Library Discovery technologies and What Can They Do?," I suggest publishers go beyond the conformance statement and do discovery auditing at five levels-- for discoverability, visibility, linkability, accessibility, and trackability-- and also across three major systems.
JULIE ZHU: So in the article, I included checklists for publishers to audit their own content, but also set up specific remediation plans in order to achieve conformance. I also included checklists to publishers, to collaborate with vendors and libraries.
JULIE ZHU: So we started doing this discovery auditing library systems in 2014, and we worked with vendors to develop auditing algorithms, created libraries-- the technical profiles-- sent auditing results to several-hundred libraries, and published configuration guides-- conducted in-person discovery workshops and online discovery webinars. All the work produced very positive results.
JULIE ZHU: Complaints about the discovery-system-related issues decreased significantly. And the usage was up, and the publisher-library engagements were strengthened. It is great to see that quite a few publishers now have designated roles in library resource discovery, and more publishers have started doing their discovery auditing.
JULIE ZHU: But publishers can only do so much. Much of the work still resides on the library side. So I proposed the idea of library responsibilities, in 2017, at the ODI Standing Committee. We formed an ODI subgroup on drafting the library conformance statement and checklist. And these were published in the 2020 version of the ODI II recommendations.
JULIE ZHU: So the recommendations covered not only how to configure the discovery service, link resolver, and authentication tools, but also collaborating, documenting, long-term planning, training, vendor communication, and more.
JULIE ZHU: So far only six libraries have published the ODI library conformance statement. So last fall, I did a limited survey of libraries' ODI conformance. Here are some selected results. I found that 83% of the libraries have designated individuals for configuring tools, but only 10% document configuration. 34% of the libraries train patrons and staff on using discovery tools, but only 7% trained staff on configuring discovery-related tools.
JULIE ZHU: About 2/3 of the libraries claim they have selected the correct databases in the discovery index, but only about 1/3 think they do well with the knowledge-based collections. So more obviously, we need more libraries to start doing their discovery auditing.
JULIE ZHU: So the continuous dialogue between the publisher and libraries turn our attention to a new area of discovery auditing-- the discovery vendor tools. The most frequent questions we get from libraries are, which database and KB collections should we select? Why couldn't we find them in the knowledge base? When there are multiple KB collections of the same or similar names, which one is correct?
JULIE ZHU: Why is the title count in this KB collection different from the title count in your KBART title list? So as we know, some publishers have several hundred KB collections. It is very difficult to monitor all. And the publishers can react to these questions passively or proactively, doing some editing in the other vendor tools.
JULIE ZHU: So we found it helpful to set up a package IDs and map them against collection IDs across major KB systems. We can use the package ID to help add, update, track collections across systems, and compare title and article counts if published vendors publish monthly reports, and give libraries more accurate guidance in selecting the collections-- especially when they change to a different KB tool.
JULIE ZHU: As we can see, discovery auditing has been evolving throughout the years. When we do discovery auditing and publish conformance statements, we do not necessarily aim for perfection, because we cannot possibly solve all the problems. It is more like a plea to all involved parties to delve into their systems and deal with issues.
JULIE ZHU: It is a pledge that we'll try our best to do our share of the work to fix the issues. And it is a process that will continue to evolve. As long as all the parties pledged to work together, we will better serve our end users in the journeys of scholarly content discovery. Thank you. I welcome questions.
JULIE ZHU:
PETER: Thank you, Julie. I want to thank all of our wonderful speakers for their presentations, and hope that everybody will join us in the live discussion part of our program, which I believe is about to begin. [MUSIC PLAYING]
Segment:0 .
PETER: Hey, welcome. As the program description outlines, this session seeks to examine how audits of many unseen factors which can impact the use of library discovery tools can help identify disconnects and incorrect information and how solutions can hopefully be forged based on what is found. We all know that publishers and libraries both want licensed content to get used.
PETER: So it's more important than ever as reliance on discovery tools increases to make sure that all is working as intended. And seeing how the factors ranging from metadata to holdings lists to links and authentication, all have to be checked. And audits of all these components can help identify what may need attention as the speakers you will hear from will outline. Today we have presenters with perspectives from both libraries, those being Northwestern University and the State Library of Ohio.
PETER: And publisher content providers, those being the IEEE and Springer Nature Publishing. And we hope you'll find much value in their experiences and their findings and their recommendations. And we'd like to start off the program with Geoff Morse from Northwestern University.
GEOFF MORSE: Thank you, Peter. Welcome. My name is Geoff Morse and I'm going to talk to you for a few minutes about the experience the Northwestern libraries had in filling out an Open Discovery conformance statement. So here's my picture and my current position.
GEOFF MORSE: I've been with the Open Discovery Initiative from NISO or ODI since 2018. So the NISO Open Discovery Initiative, to provide us with a little introduction, began in 2011 at the American Library Association's annual conference in New Orleans. At this time there were several newly emergent discovery tools that provided large aggregated index of content. However, there wasn't a lot of transparency at this point about how the content was provided, what was indexed or how conforming all of this information and material was with the standard expectations of libraries and librarians.
GEOFF MORSE: Keeping in mind that up to this point, libraries and librarians had been pretty comfortable and knowledgeable with the OPEC and knowing everything that their user could potentially search. So the Open Discovery committee was formed and made up of representatives from content providers, discovery providers and libraries to address some of these concerns. There are several overarching goals of the Open Discovery Initiative.
GEOFF MORSE: And first, really it is crucial that ways are defined for libraries to assess discovery systems and the content available within discovery systems. How libraries-- Excuse me. How librarians and users know what is included in a discovery system is essential really to anyone doing research. Secondly, it's important that processes are streamlined for content providers so they can easily work with discovery service funders to include their content in a discovery system.
GEOFF MORSE: There have also been concerns about fair linking as discovery service vendors at times are also content providers. And to address this concern around fair linking, the Open Discovery Initiative has to find models for fair linking from discovery services to the publishers of content. In the fourth overarching goal of the NISO Open Discovery Initiative or ODI is to determine what uses to systems should be collected for libraries and the content providers so we can better assess how these new tools are being used.
GEOFF MORSE: So I'll just provide a little bit of history about Northwestern's experience with discovery systems. As you can see, we've been using a discovery system Ex Lib versus Primo for a number of years. We initially started implementation back in 2008 before there was a centralized discovery index. And during our implementation, we are already thinking about adding additional records from some of our various digital collections to the discovery system.
GEOFF MORSE: And creating workflows, transforming these records from EAD and MODS into XML format. And Additionally during this period, the discovery system or since that period, the discovery system has evolved really from a secondary system kind of backing up our OPAC that was being aimed really at beginning researchers and undergraduates to really the main search system encompassing the local catalog at Northwestern as well as digital collections and the large central discovery indexed managed by Ex Libris.
GEOFF MORSE: So the NISO recommended practice document that the Open Discovery Initiative is created includes general measures as well as measures related to various aspects of configuration, communication, documentation and training. There's a checklist for discovery service providers to meet the recommended practice for Open Discovery. There's a checklist for content providers, and there's a checklist for libraries which is what I'm going to be referring to today mostly.
GEOFF MORSE: The checklist for libraries focuses on the responsibilities of libraries to configure their discovery systems to a level of optimal performance. Areas such as staff roles documentation and collaboration are covered by the checklist, which serves really as a guide to assist libraries to get the most out of their discovery system in line with the goals of the Open Discovery Initiative. In some ways, the process of doing the conformance statement checklist which was undertaken starting in May 2021 and finishing in late November, in some ways the process of doing the conformance statement has improved the mutual understanding at Northwestern amongst library staff of what others are working on.
GEOFF MORSE: It also is facilitated communication and helped place in context the work individuals are doing on the discovery system. In 2021, the library's Primo team, the team charged with overseeing the operation of our discovery system, started as I mentioned in May, started to fill in a library conformance statement or checklist which we saw in the previous screen. This was an opportune time to do this as the Primo team had recently been reconstituted with a number of new staff members.
GEOFF MORSE: So the first bullet point there relating to understanding of workflow and staff rules. The understanding of staff assignments related to the discovery system was clearly very important. While no work assignments were changed as a result of working on the conformance checklist, importantly it did become more clear who is taking responsibility for certain areas of discovery configuration.
GEOFF MORSE: The advantage of knowing who is responsible for specific roles is very helpful, particularly in the resolution of issues or problems. Cross-team communication improved as a result of this work. It's also worth noting that particularly in large organizations, and Northwestern libraries are fairly large, tracking down the right location for a role, whether the work is already being done and not documented or work that should be being done but isn't can be a challenge.
GEOFF MORSE: Large distributed organizations like ours with discovery related work happening across multiple domains and areas, often results in not everyone knowing what everyone else is doing so. And I think the conformance statement really helped us in this way as we got a better understanding of the big picture in the context of where the discovery system lay.
GEOFF MORSE: The process of filling out one of these statements required interested parties to meet and work together to establish roles and understand the larger context of the discovery system within the library. As an example, the Primo team is much more aware of the work that the electronic resources librarians do in terms of selecting databases and collections for activation and the central discovery index.
GEOFF MORSE: Library staff are better informed on potential issues with this work as well as whom to contact with very specific problems. Library staff now regularly meets monthly to discuss configuration options and new developments in the discovery system. This may seem straightforward and a pretty standard practice.
GEOFF MORSE: But it's been very helpful to regularly convene the group of people from across the libraries to work on this. These meetings enable open discussion of configuration options. To answer questions about the activation of the correct database collections, the appropriate people in the electronic resources area mentioned in the previous slide of the library had to be consulted.
GEOFF MORSE: Upon consultation, the electronic resources people could share some of the complexities behind knowing which discovery package to activate, which many of us weren't aware of just how much and how hard they had to work on that. This is facilitated better communication among members of the team and the electronic resources staff. And has enabled all of us to get a better understanding of the big picture in which the discovery system operates.
GEOFF MORSE: The Primo team is more familiar than ever now with the official documentation and will contact the vendor with questions and problems. Library staff are now more aware of whom to report problems to with the discovery system. And we've also facilitated with new online forms an easy way for staff to report any issues they see as soon as they see them in the discovery system. It results in faster resolution of the problem.
GEOFF MORSE: So again, the discovery system checklist brought a lot of these issues to the fore when we started discussing them and really helped us improve our approval processes. The conformance checklist also emphasized the importance of participating in the larger community of discovery system users to both gain information and receive assistance, as well as provide assistance to the larger community.
GEOFF MORSE: Several members belong to the Primo listserv and the library uses and contributes to an idea exchange for Primo users, where new features are suggested and acted upon sometimes by the company. And staff members also contribute to and monitor the developer's network site as well, where you can put up new features you've custom made and allow other institutions to use them or vise versa use new features that have been created by other institutions.
GEOFF MORSE: So again, the conformance checklist helps bring those issues out and emphasize them. So really the process of filling in a conformance statement has helped to highlight our need for improved documentation. Well, now without documentation of decisions that we have made, quite often that documentation has been dispersed in various places and may be lost if a staff member moves on to a new job at another institution or even another department or work area.
GEOFF MORSE: So this is one area it became clear we really have to improve. Another area for improvement was that of staff training, whether there have been sessions for staff when the discovery system was introduced or an occasional presentation on new features that really has not been a systematic training of library staff on the discovery system. And since it's our front and center discovery tool now, it seems important that we work on improving that in the coming year, which is one of our goals.
GEOFF MORSE: So just very quickly in closing, the discovery checklist from NISO Open Discovery Initiative has really helped us focus some of our discovery work and really made our discovery system work better with better communication and hopefully more optimal use for our user community. But it has also made it easier for us to work with content providers, to work with our discovery provider as well.
GEOFF MORSE: So thank you and I'm definitely open for any questions.
PETER: Thank you, Geoff. And our next speaker is David Greene from the State Library of Ohio.
DAVID GREEN: Hello, and thank you, especially thank you to the group for inviting me today. I am David Green and I am from the State Library of Ohio here in Ohio, Columbus, Ohio. And before coming to the state library, I was actually working for a non-profit consortium called OhioNet, which is a non-profit organization that supports all types of libraries across the state of Ohio be they academic or public or special.
DAVID GREEN: Whatever the case might be. And it was actually at OhioNet that I got the bulk of my experience doing what we're calling the discovery audit where I did not just one or two, but up to a dozen audits for various libraries with the state library being probably my most recent one that I have done. So how did it come to be that I managed to do so many discovery audits you might wonder.
DAVID GREEN: And a lot of it has to do with the relationship of many of the libraries to our statewide consortium called OhioLINK in the state, which has to be honest, I'm not even sure on the number at this point, but maybe 100 members academic libraries across the entire state. And it was probably, I think in 2013 that OhioLINK had negotiated a deal for many of its members to opt in to a discovery layer with EBSCO, with EBSCO's EDS at that point.
DAVID GREEN: And as you can imagine, the rollout was going to require a lot of effort. And that's where the audit came in post implementation for a number of those. So now that you know a little bit about my background, we can talk a little bit about what it looks like an audit for one of these libraries. And in every single case with every single library, these always began with discussions with library staff to come up with what we felt was the most authoritative list as possible of what our entitlements were.
DAVID GREEN: And as you can imagine, this was always easier said than done. As I mentioned, my first libraries were not just in a consortium called OhioLINK but they were actually a consortium within a consortium. So the libraries I tended to work with in the beginning were part of a consortium called the Ohio Private Academic Libraries or OPAL. And right away you can begin to see how this is going to add layers of complications when it comes to figuring out what do we own, what are our subscriptions and who do we get it from.
DAVID GREEN: As you can imagine like a library may have its local subscriptions that they purchase directly. They may have subscriptions through OhioNet. They may have subscriptions through OhioLINK. And then to make things even more complicated we also have some statewide content that's available to libraries to with statewide agreements. So picking it piece by piece, we had to figure it out who do we get it from, where do we buy it.
DAVID GREEN: And of course, as you begin to work through the and identify where these things are and what they are and you'll hear me maybe repeat this again and again but document, document, document. As a sort of extra layer to figuring out where something may have come from, one of the things that we realize right away too is that while it's easy to identify what may be current subscriptions are, we find that there's also a wealth of content that may have been perpetual access that we have purchased in the past.
DAVID GREEN: This is especially true when figuring out things that have been purchased with perpetual access at a consortium level, 1 or 2 levels up. So having a historical record of some of that stuff was incredibly helpful for identifying this content. And again, once you find these things do not just state the obvious but document, document, document. Once that preplanning is out of the way, I think that the audit itself came down to pretty much three major themes that you see here, content, access and user experience.
DAVID GREEN: This is how I see it anyway. And in terms of content, we're just talking about identifying and enabling the content itself, whether that be anything from metadata to full text. The content that you intend to be discoverable should actually be discoverable. As far as access goes, issues of whether or not you can actually get to the content. Be it navigating to it through the platform itself or navigating to a place where you can.
DAVID GREEN: And then issues of user experience as well is the content and the access configured in a way that makes sense to the user. Is it easy for the user to Find does the look and feel look good for the user? This also may sound like it's sort of a linear sort of process in terms of the-- I'll add that it's not at all linear and it is, in fact, a rather reiterated process.
DAVID GREEN: So if you haven't done one before, you can expect it to be, I don't want to say messy, we'll say reiterate. So in terms of content, we relied heavily on some of that preplanning work where we take our lists, usually broken down by the content provider and ensure that the content is configured in the discovery layer to begin with.
DAVID GREEN: And depending on the discovery layer, this might take a few shapes. So in making sure that the source of that content in EBSCO case, they call them partner databases, were enabled. At the same time discovering things that needed to be disabled or take them back out that were no longer relevant. Also finding knowledge bases and packages to configure. And while doing that, trying to discover when is the last time a knowledge base has been updated.
DAVID GREEN: If it hasn't been updated in a while, maybe that's a clue to get in touch with the vendor to talk about that. And also at the same time discovering that knowledge base packages their names frequently shift or change. And are we actually identifying the correct content? Whereas it may be something that looks similar but is actually something else.
DAVID GREEN: Choosing the right one we learned is often very difficult. And then there were some situations too where we wanted to configure content that maybe we didn't necessarily purchase or have access to but for purposes of in inter-library loan, for example. So we had not frequently, but some libraries that wanted to identify certain pieces of content make them show up in the search results.
DAVID GREEN: But only because it was highly relevant to their audience or to their students, and they wanted to at least give them the opportunity to be able to make inter-library loan request for that kind of content too. As far as access goes, we're talking about getting to the content or again, being given an opportunity to find the content on another platform.
DAVID GREEN: And as far as linking goes, I'm primarily referring to the links that are inherent to the Discovery platform. And so rather than say open resolvers. Not to say that we didn't look at open resolvers because we did and they are important. But in this case, the links that we needed to reveal were EBSCO specific. And one of the things that is not always obvious to staff and to librarians is that it's not simply enough to identify and enable the links where appropriate.
DAVID GREEN: But that there's some underlying configuration that still needs to happen. Usually this takes the shape of required parameters, for example. Usually say a customer ID, sometimes a package ID that needs to be added to the linking scheme, and without it access fails. That wasn't always obvious at first and we had found by slowly taking the process of the audit that these things were going unnoticed.
DAVID GREEN: Similarly, I found that inter-library loan links were also something that were not necessarily available by default or were never configured. Discovery platforms will usually allow you to configure either something that's baked into the platform or at least a third party ILO option. The baked-in option is what we generally did. But for all links, be they ILO or otherwise, it took some time to configure out how or figure out how to actually configure those.
DAVID GREEN: So if they were truly available for the content that we were trying to link people to. As part of that by the way, it also took a little bit of time to learn how to suppress certain links under certain circumstances so as to not overwhelm the end user. So for example, we didn't want inter-library loan links displaying on all content, only for content that wasn't available.
DAVID GREEN: Or you don't want to overwhelm your user with say 20 different options on a single record when maybe only two or three of those are the most relevant or even preferred links for your user. And you can expect that will take some time to figure out as well. And then finally with authentication there being towards the end, one of the first things that we worked through was to back up for a moment and review like IP addresses, where the IP address is still what they needed to be a configured at the platform level.
DAVID GREEN: And this often required maybe some help with some IT staff. It's one of those examples where it helps to have a working relationship with an IT department because they don't always know or think to tell library staff when there are additional IP addresses to add or that maybe sometimes they change. Similarly, all the libraries that I have worked with use EZproxy for their off campus authentication.
DAVID GREEN: So part of the audit process also required us to ensure the EZproxy was configured properly and that all platforms that were represented in the content were configured with EZproxy likewise. And then finally, something that, again that surprised us was how often people were denied access to open access content. And that was frequently because the platforms on the other end were not things that were configured with EZproxy.
DAVID GREEN: And the solution there of course, was to go back and identify what is our open access content that we have enabled. And then to turn off the proxy for those so that it wasn't blocking access for the user to get to that content. Again, something that we may not have thought about initially during the implementation phase but discovered more through reporting of users that they were not able to get that content and found a pattern there.
DAVID GREEN: So finally with user experience as you can imagine, once the agreement with OhioLINK came out, the implementation was maybe a bit rushed just given the sheer number of libraries that needed to come on board. So a lot of it came back to just the basics. Do the primary accent colors reflect my institution's brand and identity?
DAVID GREEN: Are there logos being used that reflect my institution's brand and identity? Are icons configurable? Can we change them to be more relevant to the users or to match the overall brand and identity? Are there any broken images that need to be fixed? How are they being linked to? That sort of thing. In addition to that, it also helped to review just basic information, such as contact details, hours or even simple things like navigating back to the library's main website.
DAVID GREEN: As an extension of that user experience, one of the things we also did during an audit was to review what language was being used in the discovery layer. And by this, I mean, how do we refer to things or what do we name them? What do we label them? Does the language that's being used by the interface say with limiters or otherwise actually make sense to your users?
DAVID GREEN: So for example, if your primary audience were undergraduates, a number of them might not understand what inter-library loan means. So can we change the language of that link to be something actionable instead, such as request this item or find this item, whatever the case might be. Generally, you would have flexibility in how things are labeled in a discovery layer.
DAVID GREEN: So it was just a process of trying to review what our options were and then thinking critically through like improvements to those where we thought they were necessary. And finally, some takeaways. And I'll mention here that back when the discovery layers were first being adopted, we certainly didn't know about things like ODI or conformance statements.
DAVID GREEN: We didn't necessarily have those at our disposal. And so things are definitely moving in a positive direction at this point where we do have that sort of structure in place now to help us with the self audit, especially now that I'm at the state library. These are things are becoming very useful. And we also didn't have things to such as documentation provided by vendors themselves that sometimes you will find now like say health checks or system reviews.
DAVID GREEN: And those things, I think, are finally coming out of these conversations that we're having with content providers. So important takeaways I think of this is one of the things that wasn't so obvious for us to do when we first started but is maybe something that I would insist upon now is that before we even begin an audit, you want to record a baseline so that you can see how changes may improve your discoverability and the usage of your discovery layer.
DAVID GREEN: Just as an example for that, at the state library, we had not yet set up Google Scholar and implemented that with our discovery layer. And once we did, it was only one month later that we found that 29% of the traffic that was coming to our discovery layer was now coming from Google Scholar and that was without even advertising it or even mentioning it.
DAVID GREEN: We simply just set it up and people found it. And that was a very impressive and surprising. Something else you might want to consider as you're doing this is to create a separate profile or environment if it's available. Not only will this allow you to do your work without interrupting people who are currently using it, you can sort of do your work siloed. But it also gives you a opportunity to slowly troubleshoot and configure things in a way that's maybe a little more structured.
DAVID GREEN: And then even once you do go live, becomes a nice backup that you can refer to as well. Also talk to content providers. Like I said in the beginning, we didn't have things like conformance statements to refer to or even to begin some of these conversations. But thankfully, those things exist now. But in addition to things like that, also ask your content providers how you can stay in the know for changes in content.
DAVID GREEN: So for example, do they have content alerts? Do they notify you've changed to packages to package names? If they provide those sorts of things, it's helpful to be aware of them so that you can take action when something does change. And then finally, I would also suggest that reviewing the collection development policy at your library. I think because Discovery was so new to everybody that the approach tended to be an all lean approach.
DAVID GREEN: And it was one of those things where we didn't stop to think should we, it just could we. But that may not have been the best approach in retrospect. So I think maybe a better approach moving forward in terms of doing a self audit is to consider what the collection development policy is. The content, just because we have the content, does it necessarily fit our audience and what we're trying to provide?
DAVID GREEN: Should I have limited content for example, that maybe wasn't relevant to my users? So I think that might have been something that we probably would have done differently had we considered it at the time. And with that in mind, I am happy to answer any questions about doing any number of these audits or just to talk generally about them. And once again, thank you very much.
PETER: Thank you, David. And I think everybody is seeing a real persistent theme coming through the presentations as it relates to all the different factors, any one of which can get in the way of smooth operations. So lots for us all to think about. And now to look at things from the content provider perspective, the publisher perspective. Will start off with Robert Lacey from the Springer Nature Group. Bob?
BOB BOISSY: Yes. Thank you very much, Peter. And hopefully, I can build a little bit on what you've heard so far. This is Bob Boissy speaking to you. I am Director of Account Development at Springer Nature, in the US and Canada. My relationship to the ODI Group is liaison to the ODI Working Group, from NISO Information Discovery and [INAUDIBLE].. And I know that the NISO folks would like for all of the people watching today to be thinking about what might be other work that could even enhance what we're talking about with discovery today.
BOB BOISSY: What things could possibly be built [INAUDIBLE].. So with that, I will just give you my information provider, publisher perspective on the context for discovery audits, and how we might be re-visit it. The one thing to know, coming from my point of view-- I do work for a large publishing house. And like large universities, we're lucky enough to be able to have some specialists.
BOB BOISSY: So we have folks in account development and libraries that are really specialist library support people. That means technical and promotional support. When we monitor usage-- and monitor usage is just common sense. We want to know if there's a problem before it goes too far. And so we do monitor the usage of our content with our clients.
BOB BOISSY: And the idea is that our librarians have told us if things have been going well and then they start to not go well, their first inclination is to say, that is probably a technical thing. And so we are trained to go looking for technical things first. And if it is not a technical thing, that we find no problems with the pipeline, then you might switch over to talk about some disability or other promotional type work [INAUDIBLE] on campuses.
BOB BOISSY: But the idea of a discovery audit is that we take a proactive approach to what is in fact an uncertain situation. We don't really know ahead of time how people will take up our content. So this discovery audit is to make sure and rule out technical problems. The basic elements that have been around since the mid 2000s, the publisher discovery audit, are basically that we know that we can start by looking at a library's website.
BOB BOISSY: We know that in the United States and Canada, and many other countries as well, libraries open up their catalogs, their discovery tools, their search boxes to the public. And that allows us to do a discovery audit. I have, as a matter of fact, done discovery audits with corporations as well. But that is [INAUDIBLE]. They don't open up their discovery tools to outsiders.
BOB BOISSY: Usually, I'll do that directly with them via some kind of video reading, where I ask them to search for certain things, and they come back with certain results. And we check on discovery kind of the strength of the discovery system together, sitting at our screens. So the main thing that you do during discovery audit is we will look for our own content on your systems.
BOB BOISSY: And we will do increasingly specific and use unique text elements from things like titles and abstracts and full text of content, to see if our content rises up in the results. But that's basic discovery audit. We'll do a sampling. We'll see how well things are found, and if our content is high enough in the search results that we think that makes sense for what we have searched.
BOB BOISSY: We do check Google Scholar Library Links, but a lot of libraries have that up now. We think that is an element that's important these days. We will also just be sure that you understand that we're checking book and book-chapter information as often as we check journal articles. So we do kind of an all-over-the-map type looking at the content. But today's talk is about going beyond that and revisiting the discovery of it, and perhaps amending it.
BOB BOISSY: And I've been working on this now for a while, since I joined the ODI request liaison, and encouraged library conformance statements. Perhaps out of self-interest as a publisher, I wanted these ODI statements as a starting point for a conversation. What I've realized is that it really helps to enlist the libraries earlier in discover process-- not to do it on my own, but, in fact, to work with libraries on it.
BOB BOISSY: Let them know that we're taking a quick look, that we're checking up on things. And hopefully, we're doing it together for the same reason, that we want the content to be well used. So I will check for an ODI statement. But if there is no ODI statement prepared, I've found now that I can use the checklist as a discussion, that I like to share-- so please share the checklist-- to show why I'm asking certain questions to the library.
BOB BOISSY: And I can focus on things, perhaps, that will [INAUDIBLE] to make things better. One item that I will point out is a truism, I think, that a picture is often worth a lot of words. So I have taken it up to really make sure that captions for graphics are discoverable. Many researchers are on the hunt for a graphic that they can re-use.
BOB BOISSY: And so good captioning for graphics is very, very important. And I want to make sure that our graphics are discoverable. I will also mention that KBART automation is relatively recent. And we do check for this. If it's not in place at a library, we might talk about that-- KBART automation being a way to get your holdings updated regularly with your discovery service, without you having to do that extra work manually.
BOB BOISSY: But once it's set up, it works. So in the ODI process, other NISO standards come into play-- KBART automation and KBART basic formats is part of that. I don't know if this is my personal background, or whatever, but I think it's not a bad idea to confess a little bit. It's OK to have imperfections in your discovery environment. That is expected almost.
BOB BOISSY: I will confess certain things that maybe aren't as good as I would like them to be in our metadata. With a client, might as well lay the cards on the table. But I think our metadata is largely good. And I think most publishers really work hard at it. But maybe there's imperfections [INAUDIBLE] side, binary side, or the intermediate or even a discovery layer could be some kind of work.
BOB BOISSY: And the only way to really go through that is to talk about it together. So for this purpose of this presentation, I did have a back and forth with Jeff at Northwestern about his checklist, and how they filled it out-- two pages from me and two pages back from Jeff. And that exercise, this was really very helpful [INAUDIBLE]. I sent the other folks who are presenting today a copy of that.
BOB BOISSY: So they know that we really did have quite a good conversation over the [INAUDIBLE] statement. And I was interested in the system default settings. Are they readily apparent? Are you confident that you can change them? Can you ask for changes? And the documentation of configuration decisions is very good. Whenever there's documentation, I'd love to know how public that could be-- and how open we can be and how level a playing field we can make for ourselves when it comes to sharing in relation to good discovery.
BOB BOISSY: And I like to point out to people that even though we're very much kind of a high stakeholder when it comes to discover-- in terms of if discovery doesn't work well, our mutual content does not get used. And that's our downfall. So we really are invested in it. But most of us on the publisher and aggregator and information-provider side have never actually operated in the [? Discovery ?] system, or a catalog, or a knowledge base, or a URL in a link resolver.
BOB BOISSY: So we rely on you to help us with terminology, with telling us how things work. And the library performance statement is a great step in that direction. Yeah, I am also interested in little bits and pieces of marketing. If some information resource really has a nice feature to it, could that possibly make it through the metadata and on to the discovery layer, so that that resource could show up is particularly good.
BOB BOISSY: If it weren't, if the feature was excellent, sometimes people think about, does this database [INAUDIBLE] patent data. Patent data is an extra [INAUDIBLE] across the board. And I'm just kind of throwing that out there, letting people know up front that it's some kind of feature that's available to support. Jeff and I talked about the selection of packages.
BOB BOISSY: And that's still being something of a problem. And so I [INAUDIBLE] myself package ID work, which is just starting up now. And I also want to let people know that you may talk about metadata, and you may think metadata is only [INAUDIBLE],, but not necessarily. Some metadata may be in there because it serves an electronic resource management role-- that is, if they have a role in assessment of resources.
BOB BOISSY: And if that data is needed, then we have to talk about that. Also, I just like to point out that, yes, we would like to be part of the community that talks about [INAUDIBLE] with you-- maybe with your discovery service providers-- we allowed on some generalist listservs. And we do monitor them, but we're not allowed on others, which possibly is something to look at.
BOB BOISSY: The last slide here for me is just to point out that we do have common ground. We both want licensed content to be used. We understand [INAUDIBLE] the discovery that leads to use. We want to together get the discussion going about improving metadata. It could be the handling of metadata-- the use of it-- and improving mutual support. Where can we get the [INAUDIBLE] better, to answer some of these questions that come up [INAUDIBLE] they are?
BOB BOISSY: And how can we be more transparent about the decisions we make as a provider, and you make, when it comes to discovery? And of course, my personal mission is to remind people that one good reason to keep supporting [INAUDIBLE] library [INAUDIBLE] library engineered based discovery because library referrals to resources almost always result in full text success for the end user, while public search engines deliver up the vast majority of denials.
BOB BOISSY: And I've done studies to prove this. And that means some unsuccessful outcomes [INAUDIBLE].. Even though some of those starting may be popular, they can result in bad things. So let's keep after library discovery. And that is it for me. So I will stop sharing. And let us slide over to Julie.
PETER: OK. Thank you, Bob. And as my mind is reflecting on so many key themes that have come up-- David talking about everything from using terms that our users understand to the names of packages changing. And now you've introduced this whole theme of content providers needing to get on these systems and see how their own content is working within them. That's all part of how we manage this ecosystem together.
PETER: So to tie so many of these themes together, Julie Zhu, from IEEE, who has a formidable role there in watching over their content in all the different types of discovery platforms, is going to give us a view from her perspective. Julie?
JULIE ZHU: Hi, I'm Julie Zhu, IEEE Senior Manager for Discovery Partners. I've been on the ODI Standing Committee since 2014. I will review the overall major areas for discovery auditing and point some new directions. As we all know, library discovery systems include complex data flows. And here's a flowchart of data flowing from publishers to vendors, discovery indexes, and knowledge basis, and then to libraries.
JULIE ZHU: Data flows to CrossRef and authentication systems, like it's EZproxy and [INAUDIBLE],, also have major impacts on the item's discovery, linking, and access. In 2015, I did an auditing study to examine the availability of library-subscribed content in 24 for library discovery systems.
JULIE ZHU: These 24 libraries are from different parts of the world, using four different sets of discovery service and link resolver tools. But they all subscribe to the same content. The results vary a great deal among the libraries. I found significant content gaps in the publisher systems and workflows and the vendor tools. And also, over half of the library had real problems in their discovery system and the link-resolver-tool configuration.
JULIE ZHU: So the causes for the content gaps are many. The problems can come from the publisher site, starting from the incorrect author names, or insufficient keywords, or other metadata in the author submission systems, all the way to publisher-content delivery to vendors, discovery indexes, and knowledge bases. The problems can come from the vendor side, in indexing, linking, displaying, and more.
JULIE ZHU: A considerable portion of the issues actually come from the library side. And like I mentioned, over 50% of the libraries may have issues in selecting the correct collections in the discovery index, the KB tools, and enabling remote access in various authentication systems' tools.
JULIE ZHU: So what can publishers do? They can start with the quality and delivery of their own content. The ODI content provider conformance statement is a very good start. [INAUDIBLE] was among the first publishers to publish the content provider conformance statement, in 2015, and then in 2021. In my 2018 article "Should Publishers Invest in Library Discovery technologies and What Can They Do?," I suggest publishers go beyond the conformance statement and do discovery auditing at five levels-- for discoverability, visibility, linkability, accessibility, and trackability-- and also across three major systems.
JULIE ZHU: So in the article, I included checklists for publishers to audit their own content, but also set up specific remediation plans in order to achieve conformance. I also included checklists to publishers, to collaborate with vendors and libraries.
JULIE ZHU: So we started doing this discovery auditing library systems in 2014, and we worked with vendors to develop auditing algorithms, created libraries-- the technical profiles-- sent auditing results to several-hundred libraries, and published configuration guides-- conducted in-person discovery workshops and online discovery webinars. All the work produced very positive results.
JULIE ZHU: Complaints about the discovery-system-related issues decreased significantly. And the usage was up, and the publisher-library engagements were strengthened. It is great to see that quite a few publishers now have designated roles in library resource discovery, and more publishers have started doing their discovery auditing.
JULIE ZHU: But publishers can only do so much. Much of the work still resides on the library side. So I proposed the idea of library responsibilities, in 2017, at the ODI Standing Committee. We formed an ODI subgroup on drafting the library conformance statement and checklist. And these were published in the 2020 version of the ODI II recommendations.
JULIE ZHU: So the recommendations covered not only how to configure the discovery service, link resolver, and authentication tools, but also collaborating, documenting, long-term planning, training, vendor communication, and more.
JULIE ZHU: So far only six libraries have published the ODI library conformance statement. So last fall, I did a limited survey of libraries' ODI conformance. Here are some selected results. I found that 83% of the libraries have designated individuals for configuring tools, but only 10% document configuration. 34% of the libraries train patrons and staff on using discovery tools, but only 7% trained staff on configuring discovery-related tools.
JULIE ZHU: About 2/3 of the libraries claim they have selected the correct databases in the discovery index, but only about 1/3 think they do well with the knowledge-based collections. So more obviously, we need more libraries to start doing their discovery auditing.
JULIE ZHU: So the continuous dialogue between the publisher and libraries turn our attention to a new area of discovery auditing-- the discovery vendor tools. The most frequent questions we get from libraries are, which database and KB collections should we select? Why couldn't we find them in the knowledge base? When there are multiple KB collections of the same or similar names, which one is correct?
JULIE ZHU: Why is the title count in this KB collection different from the title count in your KBART title list? So as we know, some publishers have several hundred KB collections. It is very difficult to monitor all. And the publishers can react to these questions passively or proactively, doing some editing in the other vendor tools.
JULIE ZHU: So we found it helpful to set up a package IDs and map them against collection IDs across major KB systems. We can use the package ID to help add, update, track collections across systems, and compare title and article counts if published vendors publish monthly reports, and give libraries more accurate guidance in selecting the collections-- especially when they change to a different KB tool.
JULIE ZHU: As we can see, discovery auditing has been evolving throughout the years. When we do discovery auditing and publish conformance statements, we do not necessarily aim for perfection, because we cannot possibly solve all the problems. It is more like a plea to all involved parties to delve into their systems and deal with issues.
JULIE ZHU: It is a pledge that we'll try our best to do our share of the work to fix the issues. And it is a process that will continue to evolve. As long as all the parties pledged to work together, we will better serve our end users in the journeys of scholarly content discovery. Thank you. I welcome questions.
JULIE ZHU:
PETER: Thank you, Julie. I want to thank all of our wonderful speakers for their presentations, and hope that everybody will join us in the live discussion part of our program, which I believe is about to begin. [MUSIC PLAYING]
Form Title Download
Video
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode