Cadmore media player playing video AI in the Arts, Humanities and Social Sciences: From Research to Dissemination
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
AI in the Arts, Humanities and Social Sciences: From Research to Dissemination
Description:
AI in the Arts, Humanities and Social Sciences: From Research to Dissemination
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/36ce1814-6da4-481b-a6c6-909d08cc4c16/videoscrubberimages/Scrubber_1.jpg
Duration:
T00H59M22S
Embed URL:
https://stream.cadmore.media/player/36ce1814-6da4-481b-a6c6-909d08cc4c16
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/36ce1814-6da4-481b-a6c6-909d08cc4c16/session_5a___ai_in_the_arts%2c_humanities_and_social_sciences_.mp4?sv=2019-02-02&sr=c&sig=rJKt2ZCabDWZjLIuDuzcU33eLpd6rFidh41i%2Bq71VSU%3D&st=2025-04-29T20%3A18%3A33Z&se=2025-04-29T22%3A23%3A33Z&sp=r
Upload Date:
2024-12-03T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
Good afternoon, everyone. I think we'll get going as we're at time. First of all, Thank you so much for attending our session today to talk about the topic of AI in the humanities. From research to dissemination, we hear a lot about the advantages and advances in technology in the STEM world, particularly in journals. But what about the arts, humanities and social sciences, and what about the world of books and non-journal content or scholarly publishing and research is important and helps change the world, our society, our opinions, our culture.
So we wanted to dig a little bit more into this and with our panel today, find out what they're doing in this area and in this space and hopefully give some food for thought. Before we start, though, I just want to remind you that on the app I posted about a week ago, three questions. I'm going to come back to the results of those polls, but they are still live.
If any of you want to add your thoughts, there were three questions that we posed. Do you feel you're well equipped to address the challenges and opportunities of AI as it relates to your work in HHS. Does your organization have a policy on AI and which areas of AI information do you feel you need more support on. So we'll come back to the results of that poll a little bit later in the session.
But if you wanted to add your answers, you still can do so. So let's get straight to the discussion. Bridget, if I can start with you. What has changed about AI in the last few years and how does it affect your current role and what are we doing around AI right now. Thank you for joining. Hi, everyone.
So unfortunately, there's not really a one size fits all explanation for what's happening around AI and the humanities and social sciences. But a few clear trends do kind of emerge that help explain the landscape. So we're seeing trends emerge around. Technical issue, everyone. Sorry it's.
Which in the earlier session, too. So sorry, everyone. I don't know why that's not working, but we'll figure that out. Do you want to Well, I can. I can keep talking and the visual will catch up eventually, I'm sure. So right.
So there we're seeing kind of trends emerge around what I see as five key areas, so around experimentation, operational improvements, engagement, regulation and human centricity with at least a few of those taking on a kind of uniquely cast. There's a ton of experimentation or at least planning for experimentation happening, some independently and some in collaboration with third parties.
A data point from Gartner really kind of brought the broader landscape into focus for me that I'll share with you, which is that 39% of worldwide organizations will be in the experimentation phase of Gartner's adoption curve by 2025. And just to name a couple examples in our sector. Springer Nature now has an AI research lab that explores the application of AI and machine learning in various domains, including subjects.
And they're working on projects related to content, recommendation, semantic analysis and metadata enrichment to help really kind of bolster the discoverability and accessibility of the scholarly literature. And Silverchair platform. And hosting solution for publishers has recently launched both an AI lab and an AI playground, and the AI lab was designed as a space to transparently pilot potential AI solutions.
Excuse me. Go ahead. So you can see the slide there, but not up on the screen. Perfect time for everyone to go onto the app and add your comments. While we quickly get the slides up. If you've done this before.
There we go. Thank you. You're welcome. All right. So the playground is a space for Silverchair clients to experiment with a variety of models. And so what we're seeing is publishers and vendors are very much in that kind of discovery phase.
And I'm sure we'll be seeing a lot of iteration on strategies. Sure that's OK. Thank you. So the other deck is the one that's showing because.
On this slide there. Thank you. And right. So operational improvements, we're seeing a lot of buzz around operational improvements, especially around content summarization and metadata, tagging manuscript submission and publishing workflow.
Cactus communications and Taylor and Francis recently announced paper pal preflight for editorial desk, which is an AI tool focused on three areas research, integrity, technical compliance and language quality. Sage is integrating AI into various aspects of publishing, including manuscript screening, reference, extraction and content recommendation. And Wiley also has their Rex submission tool, which uses AI to pull out things like metadata extraction to really make that author submission faster and less onerous.
I think there's a uniquely cast on what we're seeing around engagement. Most importantly is the content coming from humanities and social sciences disciplines, engaging with aspects of AI like content such as new subdisciplines, like critical AI Studies Digital humanities and AI ethics, really working to interrogate and explain what we're seeing. And there's really a desire to open up and create a dialogue.
So we're seeing some things like the lecture series recently from Palgrave MacMillan on the humanities, social sciences and AI, or the degruyter webinar with gold leaf called artificial intelligence, the great divider excuse me, divider on the regulation and policy development side, where we're seeing a lot of announcements being released around policies drafted from publishers, and we're also seeing guidance from University libraries and professional organizations like the recent research libraries guiding principles for artificial intelligence.
But as the preprint academic publisher guidelines on AI usage highlighted, quote, most publishers believed that despite their limitations, Gen AI tools can make significant contributions to academic work. But their policy is either explicitly say that they reserve the right to change their policy or left language broad or vague enough to change in this kind of fast paced environment.
There's also been kind of a big flurry of activity to update licenses. We've been seeing some big licensing deals being announced, but it will take years to out some of the implications of copyright as court cases wind through. The EU is really out in front with the Artificial Intelligence Act. But the US approach here has been more about risk management and infrastructure development.
And so as this kind of area continues to evolve, I think Roy Kaufman said it well in a scholarly kitchen post a few months ago that owners should reserve AI rights as explicitly as possible, as granularly as possible, using machine and human readable language, and should require licensees who republish their content online to do the same. And finally for human centricity. But this isn't necessarily limited to the space.
Organizations have been vocal about needing human centered approaches to ensure that design is really kind of human centered and accounting for some of the challenges like bias that we're seeing. As Brown professor Hollis Robbins pointed out in a piece in The Chronicle of Higher Education, writing is neutral and non-cultural and potentially can make facile connections. So we still need the work of the humanities to be able to add context and understanding.
And just one example where we're seeing this in action is the partnership with the National Humanities Center and Google to reimagine the training that they have in place on courses on responsible AI to bring in those approaches from those disciplines. And so new AI based tools have launched even since I made this slide a few weeks ago. It's a big area to keep up with, but just in an attempt to share a few and to group them.
I think in our area we're seeing a lot of apps and tools trying to solve pain points around areas of writing and translation, discovery and insights, ethics and integrity and research and assistance. Some of these tools actually kind of cross over into a couple of different areas here. But just to pull out a couple to explain, I won't go through all of them, but research rabbit builds itself as a Spotify for papers, and their research assistance tools includes paper recommendations, the visualization of paper topics and co-authorship, and can cluster similar papers by timeline of publication.
And so on. And then Isaac is a writing tool and it can offer multi-layered support. So things like autocomplete and reference management within the editing functionality and these things really have the potential to save researchers a lot of time on some of the more onerous parts that aren't the actual generation of their research and content.
So it'll be exciting to see how some of the opportunities there emerge. And finally, it's impossible to go through all of the implications in a short amount of time, but just to name a few. And to set up my colleagues here to dig into those in a little more detail. There's great opportunities around the research itself, new questions for researchers and new potential for interdisciplinary collaborations and new applications.
But with publishing infrastructure, there's going to be huge demands to make that machine to machine reading up to date and optimized. And the requirements for interoperability has just increased in complexity and of course for accounting for distortions that's where we're going to really need some perspectives to be able to train AI tools and to be able to call out some of these challenges. And of course, there's the tensions in governance that we need to be able to navigate.
So how can researchers use AI tools responsibly. And it's going to take time for us to really unpick all of this as publishers or libraries or vendors or other organizations that we're all learning. But researchers need support in real time. So I will pass that over now. Thank you.
Thank you so much, Brigitte. I'm Laura foster from the United States Holocaust Memorial Museum. And we have a really small publication program going the wrong way. Here we go. And when we're thinking about our publishing program, we're really looking at how I can help improve some of our operational efficiencies.
So at the moment, we're taking what is historically a print product, the Encyclopedia of camps and ghettos, 1933 to 1945. And we're trying to make it a digital product. We want to make it fully searchable. We want to make it highly accessible. We have four volumes of the encyclopedia that are already available in print and in PDF. We have three more volumes and an accompanying database that will be forthcoming at some point.
We also are updating content within volume two of the encyclopedia. So we have three stages of this product of this kind of digital launch for this product. And what we really want to make sure is that this content is highly discoverable and fully searchable. When we first started to and we want to make sure that's happening at the entry level, not just at the volume level.
When we first created this product 15 years ago, we did not have the foresight to assign keywords to each individual entry. When we created them, we created this really lovely, vast index that appears in the back of the book. And now we are in a place where we need to create this rich metadata for each of these individual entries, and including in that metadata, we have to create these keywords.
And we thought initially, great, we have these indexes, we'll go back to them and we will find the right terms for each of the entries. And then that will be the keywords we put through to the metadata. And we looked at a couple of entries and we quickly realized that just for one two page entry, we had about 65 words in our index, and 65 words is not going to translate into the number of terms we need for our metadata to make this content.
So highly discoverable. So one of we're working with Project MUSE to host this content, and we came to them and said, hey, what can we do here. And they had this great suggestion of taking our content and putting it through a closed API version of ChatGPT. So that we could narrow down the number of terms that we're using and generate some keywords. So we did this, we came back, they came back with some results.
We did 10 entries to start. And one of the things that was reassuring for us when we looked at this was that the 10 keywords that were generated out of this ChatGPT were all in our index. So there was a certain amount of comfort, I guess, for us in that. But then we also started to talk about this in house. Most of our team are, a lot of our team are researchers. So then we started to think, what else can we do here.
And we're lucky enough to be working with William, who you hear from momentarily. And he said, hey, there's also these AI tools that are specific for generating keywords keyword Bert and then one that William has developed called keyword spacy. So let's also run some tests through those and see how they compare. What are we getting out of a closed API.
ChatGPT what are we getting out of these other two. And then we can through what might be the best solution for us moving forward. Because for the Encyclopedia for volumes 1 through 4, we have thousands of entries. So it would be really time consuming for us to go through that manually and create all of those keywords. So whichever option we choose path we choose to go down, we're lucky enough we have this full team of applied researchers at the museum who create the content for the Encyclopedia.
So they will be reviewing all of those keywords to make sure that they do actually match what the entries are about. So that's a really rich discovery tool. The other things I'll mention about what we're doing when we're thinking about this is we use. We're only using closed AI tools because we don't want our content to be used to train. We also are only generating keywords using our own content to improve accuracy.
We have our applied researchers who are reviewing everything, and then these keywords are also not something that is going to go on to our website. They are things that are going to be included in metadata. And for us, that was a really important distinction because at the museum we the quality of our content and the richness of our research is so important. But when I think about our publishing program at the museum, so much of it is these historically print publications that still have a very long shelf life that people are still.
Using and people are still reading. And if we can find a way to make it them discoverable online in the digital space in an efficient way, that really starts to open up this research to a much larger audience. So, William. Thank you. So I'm going to be talking about a very specific problem specifically for information extraction.
Imagine you had a collection of texts and you wanted to identify women that appeared in these texts. This is a real world problem that we had at the Smithsonian and also at the Holocaust museum and also with a collection of documents from South Africa. And one of the things that we wanted to do is make sure that anything that we worked on and developed was reproducible with different data sets from different collections.
That's why these three different collections were chosen. And so if you look about three years ago, the different ways that you would identify gender and texts, you would use something called typically a name list. Those of you in the audience who have used these before might immediately realize the ethical problems with doing this. Namely, this would be lists of names that would map to a specific gender.
You also had ones that were backed up with statistics that would use statistics and regions to further enhance the likelihood that name mapped to a specific gender. This is highly problematic for a lot of different reasons, one of which is that names typically aren't consistent across time and space. And one of the examples that I had up here, which has now deleted use, the example of Mary went out and went out with her friend.
And if we use the term Mary, if we were to look from documents from maybe Lord of the Rings, we would realize that Mary is a gender neutral and can be applied to both. Also, people might go by different names and use different pronouns to correspond to how they self-identify. So there's a lot of issues with assigning gender and text, and it might not be as straightforward as using something as simple as a name list.
And so for this problem, we came up with gender. Spacey so Spacey is a natural language processing framework in Python. It's kind of one of the ways the industry standard for doing NLP and what gender Spacey does is it tackles the problem from a different angle. What it does is it does two other tasks to do this, one task, each of which can be done with machine learning, and then uses rules to guide the outputs from each of these different tasks.
And to assigning gender in an ethical way. So the first thing that it's done is we use named entity recognition to go through and identify all the different people in a text. Next, what we do is we look for all the different indicators in the text that can assign gender securely. So things like pronouns. So the use of her in that first example that I just gave the use of honorifics.
So maybe something like. Mrs. but we also have gender neutral honorifics and gender neutral pronouns in English. And so one of the things that it does is it flags gender specific and gender neutral pronouns honorifics. The next thing that we do is we perform a task called co-reference resolution. This is where we use if you had a nun in eighth grade like I did named Sister Catherine, you might remember diagramming out sentences.
This is essentially what we're doing here, except we're doing this with computational linguistics. What we do is we identify how words in a sentence or a document relate back to other things like antecedents or proceeds, so that you can securely link maybe a pronoun with its antecedent or the governing noun. So in this first example linking Mary with her. We then use co-reference resolution to do this task in a couple of different ways by linking these different components of the sentence together.
Next, what we do is we combine all of these approaches. And it's a little hard to read. But for those of you in the front audience, in front of the rows, you might be able to see what we've done here is we're able to assign a different kind of named entity recognition with these genders actually attached to the specific names. In other words, we've used two different components that are kept separate and then used rules to combine these things together to make sure that gender, when identified, is based securely on things that are found within the text.
In other words, working with the data and not relying on something that might be highly biased like names, lists. The other thing that we do is we also have names that are embedded within what we call spousal entities. So imagine the statement Mr and Mrs. Charles Walcot donated a bird specimen. Embedded within this is a spousal entity Mr. and Mrs. Charles Walcott.
Mrs. Walcott is entirely relegated and deleted from this entry because her husband's name Bears the most of that entity entity. In that statement. And with traditional nature, you would typically only extract the Mr Charles Walcott in this sentence. So what we've done is we've also incorporated another approach where we identify spousal entities, separate them out and use the honorifics to identify women that appeared here and separate them out as separate entities as well.
All this is being done to run across different documents in the Smithsonian, specifically regarding specimen donations and specimen findings and also across Holocaust documents such as oral testimonies and documents from South Africa, mainly from the Truth and Reconciliation Commission in the late 1990s. And early 2000. So this is an example of how we can use machine learning guided by rules and of course manual validation at the end of this process to help identify and extract different kinds of structured metadata that might be highly nested and very complex within a given document.
Let's get. Hi, everyone. My name is Rebecca dikow and I'm with Yale library. I'm the director of computational methods and data, which is a new department in the library that brings together geospatial support services, the digital humanities lab, statistical support services and research data management.
And I thought I would start by talking a little bit about the research data lifecycle. And the library is we like to say the library is the heart of the University. And also we see researchers at every part of this lifecycle. Now we know that AI tools are also kind of being incorporated at multiple parts of this life cycle. So it's quite challenging at this point, since ChatGPT became available.
Just the. How everyone now has access to AI tools in lots of different ways. And at the library we're often at the forefront seeing researchers when they're just thinking about an idea or just planning a project. And so our librarians really have to understand these tools and have a level of literacy, which is quite challenging as the tools change all the time.
So some of the things that we're doing to help people navigate computational tools in general, AI is just one of those but it's providing research consultations to students and other researchers, which are obviously central to what the library does, but really focusing on data quality. And literacy is really a lot of just about understanding the data that trained the models that you're trying to use, the data that researchers are publishing and putting out into the world, making sure it's documented in such a way that other people can use it.
We also have to talk to researchers about privacy. If they're using ChatGPT with their own data or with data that the library licenses that could be being sent back to OpenAI if they're using the free version. Just thinking about where they're putting their data and what might happen to it is really important for everyone to know. I'll go to my next slide.
So I may have more questions than answers at this point. But some of the questions that we're thinking about on the library is sort of access to AI tools. Is it equitable across campus. And I would say that it's not so as everyone here knows, that AI has been more of a part of certain disciplines for much longer than others, and certain departments at Yale will have much will have a lot of courses that potentially use AI already.
You may have students coming up in computer science and other STEM disciplines where they're building their own AI models. But then you have other disciplines, particularly in the humanities, where these tools are really much more brand new for faculty, for researchers, and yet now everyone has access to them suddenly. So we're also thinking about how the library in that way is an equalizer across disciplines.
We need to make sure we're serving everyone. We have heard from a lot of students in the humanities that they don't feel like they have data, or they don't have a data management problem. But then when you talk to them about their research and their projects for their coursework, they talk about very fundamental data management problems. They have a bunch of research materials that they've made observations on, they have notes, they have photos.
If they're looking at special collections and how to manage those is an organizational problem for them. We see that very naturally as a data management problem. But so finding the right sort of language to include everyone in these conversations is also really important. And I think this sort of messaging around learning and research, so how we're using these tools potentially to speed up work or summarize texts versus using them in a research project that might be published, and how do we help people navigate the fact that some of these models are not truly reproducible.
Potentially, if you use some of these AI models, they change all the time. They don't necessarily explain what they definitely don't explain what training data were used. So it can be difficult if you're working on a project that you plan to publish. And how can we encourage critical thinking about AI applications within the library.
I think it's really such. I'm very new to Yale. I've only been there seven months, but I think, wonderful. The library environment where we have people from all disciplines coming together to ask questions about research and data, really provides that great opportunity to think critically across the University. So some of the guidance we give to our patrons, as I mentioned, sort of describing your data that you publish so that others can use it responsibly.
And while when I was at the Smithsonian previous to coming to Yale, William and I worked along with other colleagues in our team to write data set cards for a lot of our open data sets. So these can be called data statements, data cards. They're really just describing how the data were produced, who touched it, how if it's text data, was it transcribed, was it scraped from the web. What was the sort of process for these data.
It just makes it much more usable to other people who want to use it with AI tools. And then license data. This is a really big challenge and I'm sure you're all aware of is that as Bridget mentioned, the licenses are changing or the vendors are restricting more. So what can be done with license data. How do we. Enforce that.
If it's part of an agreement that Yale has signed and the library is providing access to the data, how do we and who enforces those terms about using not using AI tools on that data. And then knowing the AI tools as much as possible. It's difficult to know everything about these tools. What are the limitations of things that we know about the models that students and other researchers are using.
What do we know about the data that the models were trained on. Can we prevent hallucinations. In a research context, this is crucial. And thinking about reproducibility. How can we ensure that if you publish your work using some of these models, that other people will be able to. Or reproduce it. That's all I have.
Thank you. Thank you to all our speakers. So you've been given a bit of a landscape of what's happening in AI and then some very specific examples of how AI tools are being used in the world. So I've got a few more questions just to follow up on that, because obviously what Rebecca has also discussed is that there are still questions to answer. But a lot of headway is being made.
So thinking about the way that you have all used AI in some capacity, I'd like to ask, what do you think the unique opportunities are specifically in HSS and what have you learned from what you have done. And maybe Bridget, you could start the conversation and then Rebecca, Laura and William can add to that. We'll come up here.
And up so you can actually see me back here. But for me, the big opportunity that comes to mind in the space is thinking about lowering that barrier to entry. And that could apply in some pretty disparate examples. But just to think of a few. I know when I was a book editor in the humanities, a big topic of conversation was there was such a demand for translating work, but there wasn't. There weren't the budgets for it.
And so academics would come to us as publishers, but we didn't have the budgets for it either most of the time. And just knowing that some of these tools, these translation tools can help to make that mean they're not necessarily going to be you're not just going to put the text into the translation tool and publish it, but it can still help to bring some of those costs down. Or at least help somebody to access that content to use in their scholarship.
And so on. But then on a different end of the spectrum, if you think about product development at publishers where there still resources, a big issue, I can do things like create a synthetic data set that might be able to help your product development that is less expensive than having to get your own data house in order and structure and do those kind of big projects. So I think there's a pretty wide range of examples where that barrier to entry could be an opportunity.
I think one of the things that was most interesting for me in the last couple of months and part of it comes out of some of the work that's being done by digital scholar scholarship Fellows at the museum is that our content, they view something like the encyclopedia as a data set. And when they're looking at that content, they want to be able to pull the data out of those entries in a really easy way.
And so it made me think about our content a little bit differently, that, Yes, we need to host it in HTML and PDF so that can be found on the web and so that people can have a really nice reading experience. But then they also need to be able to get data out of it. And that's a different format. And then how do we create those files in those formats, make people know that they're there, but then also protect them.
Because while I want to make sure that our digital scholarship Fellows have the opportunity to use that really rich data, I want to make sure I know who's using that really rich data. So that was something that I had not been thinking about before we started to have these conversations around large language models. So for me, the main thing I've thought about over the last three or four years is the way in which you always hear the phrase that AI is going to take jobs and that's not really going to happen.
What we're going to see is this hybrid ecosystem in which we have what we call human in the loop or machine in the loop, where we do our jobs with AI, essentially assisting in some of the places where it excels and then doing manual validation and manual correction. And this is a good thing. And so in a lot of different projects for things like named entity recognition, you might need to train a supervised model to do this task and that's going to take maybe months, maybe weeks.
Depends on how much data you have to collect to annotate a large amount of data that can then be used to train a model. One of the things that you can do is you can use machine learning models in the loop to help the annotation process. So that you can create gold standard data sets that have been manually validated and then train a separate model to do that task more efficiently and more accurately.
But no matter what, at the end of the day, if we're working with archives that are particularly sensitive, there's always going to be a manual validation process before data goes out to the public. Institutions are not only the keepers of knowledge, but they function as kind of an authoritative source of that data. That means that once that name is on it, once it goes out, it's not going to be easy to pull that data back and it's going to have that authoritative mark on it.
So no matter what humans are going to be used in this ecosystem to constantly validate what machines are doing, but that's a good thing. It means that we can get through the backlog of collections more effectively and easily and still have the same quality if we're doing that manual validation. Yeah, agree with what everyone has said. I would add a little bit to Bridget's point about accessibility, providing being able to give image tags and to be able to meet the accessibility standards for putting content online is a great opportunity for things like special collections, which few people get to see.
But if we can use AI and machine learning to make data more available, Yes, there has to be human in the loop. And we need to potentially have custom models in order to do that accurately and well. But it's just an opportunity that we'll never have enough staff to manually catalog things in a way to be online. So that's where I see a huge opportunity.
And I think it's interesting just listening to our panelists that we're not talking about creating content using AI. What we're talking about is enhancing the content that we already have and to make it more accessible to pull out of that. The fact that humanities has data, you just if you were in the session this morning about data in the humanities, they talked about a project where they have cataloged all of the slave trade voyages across the ocean and how many people came over in each trip and when they embarked and when they disembarked.
This is all data. It's just because it's not an equation or a gel blot or a big table. It doesn't mean it's not data. And so I think we have to change the narrative. Also, when we talk about data and accessibility in the humanities and arts, because it's about your perspective. And I think that's why I think we need to have these conversations to think about this in a different way.
So a lot of what we've talked about is enhancing the content we already have and making it available to a wider audience. And I think that's a really powerful point to take away. But that also brings me on to another question that I'd like to ask our panelists, which is around ethics. And so we've talked a bit about opportunities, but what do you think are some of the challenges, especially around the ethics of using AI, especially with potentially some sensitive material that is again, it's opinions, it's people, it's about lives, it's about culture.
And how would you how would you deal with some of those ethical challenges. Do you maybe Rebecca, do you want to say anything on that first. Sure and I think it's come up a lot in what William talked about already. I think it's really just being aware of the limitations of the models you're using and what data they were trained on.
So we there's a lot of garbage in ChatGPT, so we need to be aware of is there potentially a list of terms, we would not want put on our data. We could customize a model to make sure that those terms never get assigned to our data. It's really about bringing the content experts in early, I think in identifying what they're looking for in making data more available and what are the hot button areas where ethically there will be challenges.
I can pretty much guarantee that they'll know what those are already. And so really just reverse engineering the computation to make that happen in a way that people are comfortable with. And having a disclaimer potentially things are put online, having a feedback mechanism for community members to say, this doesn't look right and to really be responsive, then I think that's the best we can do.
It's not perfect, but Yeah. So I'll talk about two different aspects of biases one explicit and one implicit. So from South Africa, we have a large collection of documents from the Truth and Reconciliation Commission which detail human rights violations essentially over the course of many decades in South Africa. One of the things that we could work into the model is we could work into the model specific labels to extract.
So we could identify perpetrators of violence, maybe the type of violence that occurred and the victims of violence. Just because you can work a label into a model doesn't mean that you should. What's the natural downstream effect of that. Generate a list that's not going to be manually validated, potentially that can identify a whole bunch of perpetrators. What if 5% of those perpetrators are wrong, which is a very likely outcome that leads to a whole bunch of not only ethical problems, but legal problems as well.
And so, like Rebecca said, working with a team of content experts and spending a lot of time coming up with a controlled vocabulary of what can and what should be worked into a model is fundamental. Another bias is more implicit, and it's not necessarily something you would notice on the surface. If you're using large language models all the time. And that's going to be the effect of low resource languages or languages that don't have as many texts digitized and publicly available.
And this is another kind of bias. This is a bias where models don't perform as well on these kinds of languages. So in South Africa, you have a collection of Bantu languages so Zosia, Zulu, et cetera. These are low resource languages because there's not a lot of digitized texts on them. So what are some ways you can overcome those biases. Well, one of the things you can do is you can go out and you can collect or digitize a lot more texts in these areas and fine tune open source large language models on this data so that those who maybe are researchers or human rights lawyers might have the same equitable access to doing tasks like named entity recognition on documents from those languages or those language groups.
So those are two different kinds of biases. Thank you. I'll see. One of the challenges that we've run into is when we were looking and working through this metadata issue with the encyclopadie idea and we still kind of are one of the other things that we considered using an API for was to create abstracts, which are another really valuable part of metadata.
But we got into it and we realized, no, I don't think that's someplace we're comfortable with. I don't think we're comfortable with content being generated through an API and then attached to our own original research. And that was just I don't think that until we saw those abstracts literally in black and white that we realized that like no, we're uncomfortable with this and that keywords are probably going to be enough for our purposes.
But I think it all links back to this common theme of even though these are really useful tools, you still need content experts who are reviewing this content. And when it comes to I think for us, when it comes to creating that content, that is still a human who is a researcher and a writer and is really skilled at doing those jobs and communicating that the research that they've done.
And so I'm now going to go to I know what you've all been waiting for is the results of the poll. So we have had some more responses since we've started. So the three questions that we posed was, do you feel Well equipped to address the challenges and opportunities of AI as it relates to your work in NHS. The majority. So we had 26 people respond and the majority said no. So that's quite definitive.
Does your organization have an AI policy. 38 people responded and the majority said Yes, but 14 people said no. And then I think one which I thought was the most interesting. Which areas of AI information do you need more support on. We had 32 people respond to this. The options were policy and publishing ethics. Six people said Yes.
Application to publishing only two people. Ethical considerations. Six people resources four people. But by and large, the one that had the most people vote for it was legal issues like copyright and licensing, where it's an obvious indication, just even from this small sample, that people are still needing support around how they can use AI, AI tools, AI in their day to day work, what the implications are of that, what they can and cannot do, and actually how do you actually implement that into your day to day work.
And hopefully today you've heard a few case studies of how it has actually been worked into projects and work that people are doing but hasn't changed. The root of the goal of what they're trying to achieve. It's just helped to get there. But it feels like there is still a lot of work to do on more practical implementation of these tools. And so that brings us on to coming towards the end of our talk but opening up to questions from all of you for anything that you've heard today.
Any other comments you may have, any thoughts. We're very keen to keep this conversation going around the humanities and social sciences, so we'd be very keen to hear more from anything that you all have been working on and would like to comment on. Great I'll come over there. Hi, Thanks so much. My name is Rebecca.
I am a colleague of. I'm going to start over. I just lost my train of thought. I'm from Wiley. I'm a senior design strategy manager and I really appreciate all of the commentary that was provided here today on the panel. What's very interesting and very timely is that OpenAI actually released ChatGPT yesterday.
So a lot of what we're talking about regarding responsible AI on campus and within on campus entities might find that as a solution. The jury is still out. As an adjunct professor, I have struggled with students using ChatGPT to generate full on research proposals and papers, so I'm interested in investigating that further. But my question is and I'm not sure if this is relevant to this audience, but I'm curious if you have interfaced with any related funding proposals or grants that have mentioned any type of cost or cost structure for integrating generative AI resources into a project.
And if you have seen that, what does that look like. Good question. Does anyone want to take that first. I can answer. It just finished a grant application that did this. OK I think I have a loud voice. I'm OK. So yeah, so one of the things that we did was OpenAI or cloud or any other service will oftentimes in the number of 1 million tokens give you the current number.
And that unless it's Google's Gemini, it won't fluctuate too much, which just went up, I think double in cost last week. So most of the time these costs are somewhat reliable. If anything, they're going to go down, especially as new models come out. So one of the big challenges is that if you're in the humanities in any age grant, let's say you start putting start drafting it, you're not going to get the grant for probably six months to a year.
Once you get the funds, maybe another three months, once you get to the stage where you're ready to start using the API. Another three months and that time maybe three new versions of GPT come out and the cost has either gone up to use the better quality one or gone down to use the ones where you started at when you started writing the grant. So what I would recommend doing what we did was we just.
Build it essentially with what we estimated to be the number of uses in the millions of tokens. This was for generating synthetic data to fine tune a large language model and we build it according to that. And we made a note that it was an estimate and that we were going to pull from the reserve funds in case we needed to use more.
Thank you. Any other questions. Hi, I may. I'm here from Emory University, where I work with humanities researchers who are publishing books. And so I'm not sure if this is a question or a train of thought. So apologies in advance, but we hear so much about humans will validate, humans will validate.
We'll just the machines will do it, the humans will validate. And I worry a little bit about whether we are underestimating how that will work and whether people might need additional training if their job is then as a validator and not as a writer. And so I wonder if any of you have thoughts about that. And then kind of connected to that. We also I mean, I've heard that in this session being mentioned. Well, we're not it's not the machine is not creating it.
We're not having the creating the content. And obviously, we can draw lines in different places. But anyone who is in the panel yesterday where Springer was like, here's our book designer, we can make whole monographs. I think that there is a we're moving in the direction of that being something that some publishers are OK with. And so I kind of wonder yeah, whether that is just going to be something you anticipate is a line.
We just need to keep discussing where it is or what it means to co-produce content, especially because the researchers I work with where they use it is in drafting. So that's the start already. And so I think that that's a question, I'd be interested to hear what you all think. Thanks so much. No, Thank you.
Very good question. So maybe I can start with this one in terms of where we are at Bloomsbury. So I think you're absolutely right. I think there is an element of training that is needed. I think and we'd be silly to think that you can just have this as part of your day to day job. So what we're doing is we've created an internal steering group where monitoring the use of ChatGPT and other tools.
We're creating guidelines for how our authors can use it, which is acceptable for us. And we're drawing a line in the sand. But you're very right. I think that line will continue to change. I don't think it's set in stone. I think as we evolve I mean, as we've mentioned, in two weeks time, there's probably going to be a newer version of something else that's come out.
So this is going to continue to evolve. And I think we have to be flexible and nimble and we have to adapt with the way that the world of research is adapting. But I still think that there is still a line. And I think each publisher and each institution has to make up their own mind about where that is drawn. I used to work at Springer Nature, so I've been in the journal world and now moving into books, and so I'm very familiar with both sides of the story.
And I think each publisher has to make up their own decision about where they're comfortable being. I think there may well be a time where we're using AI to create summaries of content, perhaps, or to create different types of content. So I'm not saying that will never happen, but I think it's a conversation that we have to keep discussing because I think it's about the ethics. I think it's about the validation.
I think it's about where your values are and what you feel. For us, original and excellence in work is first and foremost. So all the questions around authorship and copyright and whether it's the version of truth and whose truth are really important, and I don't know that we have the answers, but it's a work in progress. And I think everyone's trying to figure that out in different ways.
So I don't think there is a right or a wrong way. But I think keep having the conversations. And I think that line will continue to move. But I think also that as an industry, we need to share what we're doing and actually be a little bit more open about that as well, because that's how we will learn and develop collectively, because fundamentally we're all about the research and disseminating that research and making it available to as many people as possible.
So if we keep that at our core, then I think we can work more together. I don't know if any of you want to add something, just to speak a little bit to the second part of your question. The preprint that I referenced earlier in my slides, it did an analysis of the publisher policies on authorship. And it was clear that publishers were drawing that line.
And I think from the Committee on publishing ethics. The difference is like, yes, you can generate content, but GPT or AI tools can't take responsibility for that content. And so that's why we're seeing those policies. So even though there's these kind of experiments that are getting buzz like the book that was written by AI, by Springer Nature, as far as those policies, I don't see any indication that that's necessarily going to change.
But my colleagues might have some thoughts on the first part of your question. One of the things when we started to think about how people are going to check these keywords because that's going to now need to be a part of our process over, I think the next six months for thousands of entries is that it's a resource issue. Like we've now done this and we've gotten further than we would have gotten without this tool.
But now we've created more work that needs to be done and we now need to get it done very quickly in order to get this product launched. And I think that's something what we're starting to grapple with a little bit. And one of the things to your second question, one of the things that's kind of resonating from in my mind from that conversation yesterday and with an AI generated book is what is the role of the researcher.
Is the role of the researcher to do the research or is the role of the researcher to do the research and be able to communicate that research. Because when you have an AI suddenly step in to do the writing and then that removes part of that role of the researcher. And so it reshapes that and I think it reshapes it in a way that is I mean, I can speak for where I'm coming from.
It reshapes it in a way that, for humanities I'm not really comfortable. That's part of this process. And that's what makes it unique and that's what makes it a really wonderful product. And that's also how you learn more about the research you've done is through that writing process. So you start to remove that and I worry it starts to unravel a larger part of the academic ecosystem.
Yeah, and sorry, I'll take it one more step further. I think even more so for the Humanities because especially if you went to Karen Wolfe's the talk they did yesterday about the crisis in the humanities and what are the really big skill sets that you come out of college, undergraduate or graduate or PhD from a humanities major. It is writing clearly, it is doing good research. And those are skills that you use to then later further your career.
And so if we start to erode the value of those, it starts to really impact that in a much larger way. Yeah, I totally agree. Anyone else. I was going to say, there's really two things that lead to a better outcome with human in the loop. The first one is going to be a well defined schema and that's going to look like the things that you want to extract.
So the labels a description that's very clearly defined so that they can reference back to it and also examples that they can go to. And the second thing that you include once you've seen a lot of different annotators do things is adding an option for I don't know or this needs to be this needs to be discussed just by giving annotators that out, you're going to remove a lot of really bad annotations almost immediately.
And make it very clear to your annotators if you're not absolutely certain, hit that button, it's OK, and then you can convene as a group and discuss those more challenging issues. And those are going to be the ones that help you rethink the schema and rethink how you're going to structure the project. And especially earlier on when you're doing the annotating with the human in the loop, having regular meetings to discuss what you're finding, what's not really aligning and adjusting early on, so you don't have to go back and redo everything.
Any other questions. Yes hey. Hi my name is Avi. So when we talk about the LMS or tools for publishing, the one question that comes to me is, so all these elements are trained to September 2022 data or not the updated data. So when we use this LLMs for the research integrity or peer review management, if you want to publish in this domain or stem domain.
So do you think. Should we check the integrity of its outcome because as it's not trained on updated data. So because that could be a problem for a peer review management or a research integrity. So any thoughts on that. Yeah, I can. That's a huge problem.
And I think that's not at all solved how to deal with that. I think particularly a lot of students are using ChatGPT to generate code and it works pretty well. If you're a new coder and you just want to make a plot of something great, but if you're trying to follow along, like maybe you're doing it in two windows and also looking at Stack Overflow and that you might not get the same answer because ChatGPT is out of date.
And that is a. That a. It challenging. So you really have to know that going in that, OK, I can maybe use this as a template, but I'm going to have to modify it if I'm actually going to use it for a research product. And that I think is really important to convey to New users because you can't really take it at face value.
I mean, you definitely can take it at face value what you get out of it. And there's two ways to really overcome limited knowledge. One is going to be hooking it up to a knowledge base in some capacity. Oftentimes, this is going to be a vector database and you can use what's called retrieval augmented generation or RAG, which is going to basically perform semantic search first based on some kind of query, go out and find the relevant documents or the primary sources, and then use those as what we call a context to improve and give the knowledge necessary to answer whatever that question might be.
This is really good, not only with outdated material, but with material that's really highly domain specific. So maybe a collection of oral history testimonies or oral testimonies from the Holocaust that weren't worked into the model that have individuals who only appear in this one document and all the collection of public documents. So that's one that's useful. Another thing, another approach is I lost my train of thought.
There is a second one. I promise you, when it comes to me, I'll say it. Sorry, I remembered. The other option is to fine tune an open source language model on your specific domain texts. This is really useful if you're working with a very specific language or a highly specialized collection of documents, like something from maybe the medical industry or a particular area of historical research, like maybe early medieval Europe.
We're at time now unless there was any other questions or comments. Yeah, sure. Thank you so much. This is for Laura. I'm Nia for Brown University library. And I was just wondering, have you had discussions or are you already planning to indicate to your users or readers in some way that the keywords were generated with AI.
Yeah, that is part of our conversation. And I think one of the things that makes the way that we're using it a little different is that if we were putting those keywords on our site, we could very clearly say these keywords were generated using AI and checked by a researcher. But because they're going into our metadata and so they aren't available to the standard user, they're going out as this upload to these discovery partners.
I think we're going to need to work on what does that look like and what's necessary for us to put in there as a caveat. But I'm also happy to continue the conversation as well. So I'm going to close the session. Thank you again very much, everyone, for coming. Thank you to our panelists. And if anyone does want to continue the conversations, we're really happy to speak after the session. Thank you so much.
Segment:0 .
Good afternoon, everyone. I think we'll get going as we're at time. First of all, Thank you so much for attending our session today to talk about the topic of AI in the humanities. From research to dissemination, we hear a lot about the advantages and advances in technology in the STEM world, particularly in journals. But what about the arts, humanities and social sciences, and what about the world of books and non-journal content or scholarly publishing and research is important and helps change the world, our society, our opinions, our culture.
So we wanted to dig a little bit more into this and with our panel today, find out what they're doing in this area and in this space and hopefully give some food for thought. Before we start, though, I just want to remind you that on the app I posted about a week ago, three questions. I'm going to come back to the results of those polls, but they are still live.
If any of you want to add your thoughts, there were three questions that we posed. Do you feel you're well equipped to address the challenges and opportunities of AI as it relates to your work in HHS. Does your organization have a policy on AI and which areas of AI information do you feel you need more support on. So we'll come back to the results of that poll a little bit later in the session.
But if you wanted to add your answers, you still can do so. So let's get straight to the discussion. Bridget, if I can start with you. What has changed about AI in the last few years and how does it affect your current role and what are we doing around AI right now. Thank you for joining. Hi, everyone.
So unfortunately, there's not really a one size fits all explanation for what's happening around AI and the humanities and social sciences. But a few clear trends do kind of emerge that help explain the landscape. So we're seeing trends emerge around. Technical issue, everyone. Sorry it's.
Which in the earlier session, too. So sorry, everyone. I don't know why that's not working, but we'll figure that out. Do you want to Well, I can. I can keep talking and the visual will catch up eventually, I'm sure. So right.
So there we're seeing kind of trends emerge around what I see as five key areas, so around experimentation, operational improvements, engagement, regulation and human centricity with at least a few of those taking on a kind of uniquely cast. There's a ton of experimentation or at least planning for experimentation happening, some independently and some in collaboration with third parties.
A data point from Gartner really kind of brought the broader landscape into focus for me that I'll share with you, which is that 39% of worldwide organizations will be in the experimentation phase of Gartner's adoption curve by 2025. And just to name a couple examples in our sector. Springer Nature now has an AI research lab that explores the application of AI and machine learning in various domains, including subjects.
And they're working on projects related to content, recommendation, semantic analysis and metadata enrichment to help really kind of bolster the discoverability and accessibility of the scholarly literature. And Silverchair platform. And hosting solution for publishers has recently launched both an AI lab and an AI playground, and the AI lab was designed as a space to transparently pilot potential AI solutions.
Excuse me. Go ahead. So you can see the slide there, but not up on the screen. Perfect time for everyone to go onto the app and add your comments. While we quickly get the slides up. If you've done this before.
There we go. Thank you. You're welcome. All right. So the playground is a space for Silverchair clients to experiment with a variety of models. And so what we're seeing is publishers and vendors are very much in that kind of discovery phase.
And I'm sure we'll be seeing a lot of iteration on strategies. Sure that's OK. Thank you. So the other deck is the one that's showing because.
On this slide there. Thank you. And right. So operational improvements, we're seeing a lot of buzz around operational improvements, especially around content summarization and metadata, tagging manuscript submission and publishing workflow.
Cactus communications and Taylor and Francis recently announced paper pal preflight for editorial desk, which is an AI tool focused on three areas research, integrity, technical compliance and language quality. Sage is integrating AI into various aspects of publishing, including manuscript screening, reference, extraction and content recommendation. And Wiley also has their Rex submission tool, which uses AI to pull out things like metadata extraction to really make that author submission faster and less onerous.
I think there's a uniquely cast on what we're seeing around engagement. Most importantly is the content coming from humanities and social sciences disciplines, engaging with aspects of AI like content such as new subdisciplines, like critical AI Studies Digital humanities and AI ethics, really working to interrogate and explain what we're seeing. And there's really a desire to open up and create a dialogue.
So we're seeing some things like the lecture series recently from Palgrave MacMillan on the humanities, social sciences and AI, or the degruyter webinar with gold leaf called artificial intelligence, the great divider excuse me, divider on the regulation and policy development side, where we're seeing a lot of announcements being released around policies drafted from publishers, and we're also seeing guidance from University libraries and professional organizations like the recent research libraries guiding principles for artificial intelligence.
But as the preprint academic publisher guidelines on AI usage highlighted, quote, most publishers believed that despite their limitations, Gen AI tools can make significant contributions to academic work. But their policy is either explicitly say that they reserve the right to change their policy or left language broad or vague enough to change in this kind of fast paced environment.
There's also been kind of a big flurry of activity to update licenses. We've been seeing some big licensing deals being announced, but it will take years to out some of the implications of copyright as court cases wind through. The EU is really out in front with the Artificial Intelligence Act. But the US approach here has been more about risk management and infrastructure development.
And so as this kind of area continues to evolve, I think Roy Kaufman said it well in a scholarly kitchen post a few months ago that owners should reserve AI rights as explicitly as possible, as granularly as possible, using machine and human readable language, and should require licensees who republish their content online to do the same. And finally for human centricity. But this isn't necessarily limited to the space.
Organizations have been vocal about needing human centered approaches to ensure that design is really kind of human centered and accounting for some of the challenges like bias that we're seeing. As Brown professor Hollis Robbins pointed out in a piece in The Chronicle of Higher Education, writing is neutral and non-cultural and potentially can make facile connections. So we still need the work of the humanities to be able to add context and understanding.
And just one example where we're seeing this in action is the partnership with the National Humanities Center and Google to reimagine the training that they have in place on courses on responsible AI to bring in those approaches from those disciplines. And so new AI based tools have launched even since I made this slide a few weeks ago. It's a big area to keep up with, but just in an attempt to share a few and to group them.
I think in our area we're seeing a lot of apps and tools trying to solve pain points around areas of writing and translation, discovery and insights, ethics and integrity and research and assistance. Some of these tools actually kind of cross over into a couple of different areas here. But just to pull out a couple to explain, I won't go through all of them, but research rabbit builds itself as a Spotify for papers, and their research assistance tools includes paper recommendations, the visualization of paper topics and co-authorship, and can cluster similar papers by timeline of publication.
And so on. And then Isaac is a writing tool and it can offer multi-layered support. So things like autocomplete and reference management within the editing functionality and these things really have the potential to save researchers a lot of time on some of the more onerous parts that aren't the actual generation of their research and content.
So it'll be exciting to see how some of the opportunities there emerge. And finally, it's impossible to go through all of the implications in a short amount of time, but just to name a few. And to set up my colleagues here to dig into those in a little more detail. There's great opportunities around the research itself, new questions for researchers and new potential for interdisciplinary collaborations and new applications.
But with publishing infrastructure, there's going to be huge demands to make that machine to machine reading up to date and optimized. And the requirements for interoperability has just increased in complexity and of course for accounting for distortions that's where we're going to really need some perspectives to be able to train AI tools and to be able to call out some of these challenges. And of course, there's the tensions in governance that we need to be able to navigate.
So how can researchers use AI tools responsibly. And it's going to take time for us to really unpick all of this as publishers or libraries or vendors or other organizations that we're all learning. But researchers need support in real time. So I will pass that over now. Thank you.
Thank you so much, Brigitte. I'm Laura foster from the United States Holocaust Memorial Museum. And we have a really small publication program going the wrong way. Here we go. And when we're thinking about our publishing program, we're really looking at how I can help improve some of our operational efficiencies.
So at the moment, we're taking what is historically a print product, the Encyclopedia of camps and ghettos, 1933 to 1945. And we're trying to make it a digital product. We want to make it fully searchable. We want to make it highly accessible. We have four volumes of the encyclopedia that are already available in print and in PDF. We have three more volumes and an accompanying database that will be forthcoming at some point.
We also are updating content within volume two of the encyclopedia. So we have three stages of this product of this kind of digital launch for this product. And what we really want to make sure is that this content is highly discoverable and fully searchable. When we first started to and we want to make sure that's happening at the entry level, not just at the volume level.
When we first created this product 15 years ago, we did not have the foresight to assign keywords to each individual entry. When we created them, we created this really lovely, vast index that appears in the back of the book. And now we are in a place where we need to create this rich metadata for each of these individual entries, and including in that metadata, we have to create these keywords.
And we thought initially, great, we have these indexes, we'll go back to them and we will find the right terms for each of the entries. And then that will be the keywords we put through to the metadata. And we looked at a couple of entries and we quickly realized that just for one two page entry, we had about 65 words in our index, and 65 words is not going to translate into the number of terms we need for our metadata to make this content.
So highly discoverable. So one of we're working with Project MUSE to host this content, and we came to them and said, hey, what can we do here. And they had this great suggestion of taking our content and putting it through a closed API version of ChatGPT. So that we could narrow down the number of terms that we're using and generate some keywords. So we did this, we came back, they came back with some results.
We did 10 entries to start. And one of the things that was reassuring for us when we looked at this was that the 10 keywords that were generated out of this ChatGPT were all in our index. So there was a certain amount of comfort, I guess, for us in that. But then we also started to talk about this in house. Most of our team are, a lot of our team are researchers. So then we started to think, what else can we do here.
And we're lucky enough to be working with William, who you hear from momentarily. And he said, hey, there's also these AI tools that are specific for generating keywords keyword Bert and then one that William has developed called keyword spacy. So let's also run some tests through those and see how they compare. What are we getting out of a closed API.
ChatGPT what are we getting out of these other two. And then we can through what might be the best solution for us moving forward. Because for the Encyclopedia for volumes 1 through 4, we have thousands of entries. So it would be really time consuming for us to go through that manually and create all of those keywords. So whichever option we choose path we choose to go down, we're lucky enough we have this full team of applied researchers at the museum who create the content for the Encyclopedia.
So they will be reviewing all of those keywords to make sure that they do actually match what the entries are about. So that's a really rich discovery tool. The other things I'll mention about what we're doing when we're thinking about this is we use. We're only using closed AI tools because we don't want our content to be used to train. We also are only generating keywords using our own content to improve accuracy.
We have our applied researchers who are reviewing everything, and then these keywords are also not something that is going to go on to our website. They are things that are going to be included in metadata. And for us, that was a really important distinction because at the museum we the quality of our content and the richness of our research is so important. But when I think about our publishing program at the museum, so much of it is these historically print publications that still have a very long shelf life that people are still.
Using and people are still reading. And if we can find a way to make it them discoverable online in the digital space in an efficient way, that really starts to open up this research to a much larger audience. So, William. Thank you. So I'm going to be talking about a very specific problem specifically for information extraction.
Imagine you had a collection of texts and you wanted to identify women that appeared in these texts. This is a real world problem that we had at the Smithsonian and also at the Holocaust museum and also with a collection of documents from South Africa. And one of the things that we wanted to do is make sure that anything that we worked on and developed was reproducible with different data sets from different collections.
That's why these three different collections were chosen. And so if you look about three years ago, the different ways that you would identify gender and texts, you would use something called typically a name list. Those of you in the audience who have used these before might immediately realize the ethical problems with doing this. Namely, this would be lists of names that would map to a specific gender.
You also had ones that were backed up with statistics that would use statistics and regions to further enhance the likelihood that name mapped to a specific gender. This is highly problematic for a lot of different reasons, one of which is that names typically aren't consistent across time and space. And one of the examples that I had up here, which has now deleted use, the example of Mary went out and went out with her friend.
And if we use the term Mary, if we were to look from documents from maybe Lord of the Rings, we would realize that Mary is a gender neutral and can be applied to both. Also, people might go by different names and use different pronouns to correspond to how they self-identify. So there's a lot of issues with assigning gender and text, and it might not be as straightforward as using something as simple as a name list.
And so for this problem, we came up with gender. Spacey so Spacey is a natural language processing framework in Python. It's kind of one of the ways the industry standard for doing NLP and what gender Spacey does is it tackles the problem from a different angle. What it does is it does two other tasks to do this, one task, each of which can be done with machine learning, and then uses rules to guide the outputs from each of these different tasks.
And to assigning gender in an ethical way. So the first thing that it's done is we use named entity recognition to go through and identify all the different people in a text. Next, what we do is we look for all the different indicators in the text that can assign gender securely. So things like pronouns. So the use of her in that first example that I just gave the use of honorifics.
So maybe something like. Mrs. but we also have gender neutral honorifics and gender neutral pronouns in English. And so one of the things that it does is it flags gender specific and gender neutral pronouns honorifics. The next thing that we do is we perform a task called co-reference resolution. This is where we use if you had a nun in eighth grade like I did named Sister Catherine, you might remember diagramming out sentences.
This is essentially what we're doing here, except we're doing this with computational linguistics. What we do is we identify how words in a sentence or a document relate back to other things like antecedents or proceeds, so that you can securely link maybe a pronoun with its antecedent or the governing noun. So in this first example linking Mary with her. We then use co-reference resolution to do this task in a couple of different ways by linking these different components of the sentence together.
Next, what we do is we combine all of these approaches. And it's a little hard to read. But for those of you in the front audience, in front of the rows, you might be able to see what we've done here is we're able to assign a different kind of named entity recognition with these genders actually attached to the specific names. In other words, we've used two different components that are kept separate and then used rules to combine these things together to make sure that gender, when identified, is based securely on things that are found within the text.
In other words, working with the data and not relying on something that might be highly biased like names, lists. The other thing that we do is we also have names that are embedded within what we call spousal entities. So imagine the statement Mr and Mrs. Charles Walcot donated a bird specimen. Embedded within this is a spousal entity Mr. and Mrs. Charles Walcott.
Mrs. Walcott is entirely relegated and deleted from this entry because her husband's name Bears the most of that entity entity. In that statement. And with traditional nature, you would typically only extract the Mr Charles Walcott in this sentence. So what we've done is we've also incorporated another approach where we identify spousal entities, separate them out and use the honorifics to identify women that appeared here and separate them out as separate entities as well.
All this is being done to run across different documents in the Smithsonian, specifically regarding specimen donations and specimen findings and also across Holocaust documents such as oral testimonies and documents from South Africa, mainly from the Truth and Reconciliation Commission in the late 1990s. And early 2000. So this is an example of how we can use machine learning guided by rules and of course manual validation at the end of this process to help identify and extract different kinds of structured metadata that might be highly nested and very complex within a given document.
Let's get. Hi, everyone. My name is Rebecca dikow and I'm with Yale library. I'm the director of computational methods and data, which is a new department in the library that brings together geospatial support services, the digital humanities lab, statistical support services and research data management.
And I thought I would start by talking a little bit about the research data lifecycle. And the library is we like to say the library is the heart of the University. And also we see researchers at every part of this lifecycle. Now we know that AI tools are also kind of being incorporated at multiple parts of this life cycle. So it's quite challenging at this point, since ChatGPT became available.
Just the. How everyone now has access to AI tools in lots of different ways. And at the library we're often at the forefront seeing researchers when they're just thinking about an idea or just planning a project. And so our librarians really have to understand these tools and have a level of literacy, which is quite challenging as the tools change all the time.
So some of the things that we're doing to help people navigate computational tools in general, AI is just one of those but it's providing research consultations to students and other researchers, which are obviously central to what the library does, but really focusing on data quality. And literacy is really a lot of just about understanding the data that trained the models that you're trying to use, the data that researchers are publishing and putting out into the world, making sure it's documented in such a way that other people can use it.
We also have to talk to researchers about privacy. If they're using ChatGPT with their own data or with data that the library licenses that could be being sent back to OpenAI if they're using the free version. Just thinking about where they're putting their data and what might happen to it is really important for everyone to know. I'll go to my next slide.
So I may have more questions than answers at this point. But some of the questions that we're thinking about on the library is sort of access to AI tools. Is it equitable across campus. And I would say that it's not so as everyone here knows, that AI has been more of a part of certain disciplines for much longer than others, and certain departments at Yale will have much will have a lot of courses that potentially use AI already.
You may have students coming up in computer science and other STEM disciplines where they're building their own AI models. But then you have other disciplines, particularly in the humanities, where these tools are really much more brand new for faculty, for researchers, and yet now everyone has access to them suddenly. So we're also thinking about how the library in that way is an equalizer across disciplines.
We need to make sure we're serving everyone. We have heard from a lot of students in the humanities that they don't feel like they have data, or they don't have a data management problem. But then when you talk to them about their research and their projects for their coursework, they talk about very fundamental data management problems. They have a bunch of research materials that they've made observations on, they have notes, they have photos.
If they're looking at special collections and how to manage those is an organizational problem for them. We see that very naturally as a data management problem. But so finding the right sort of language to include everyone in these conversations is also really important. And I think this sort of messaging around learning and research, so how we're using these tools potentially to speed up work or summarize texts versus using them in a research project that might be published, and how do we help people navigate the fact that some of these models are not truly reproducible.
Potentially, if you use some of these AI models, they change all the time. They don't necessarily explain what they definitely don't explain what training data were used. So it can be difficult if you're working on a project that you plan to publish. And how can we encourage critical thinking about AI applications within the library.
I think it's really such. I'm very new to Yale. I've only been there seven months, but I think, wonderful. The library environment where we have people from all disciplines coming together to ask questions about research and data, really provides that great opportunity to think critically across the University. So some of the guidance we give to our patrons, as I mentioned, sort of describing your data that you publish so that others can use it responsibly.
And while when I was at the Smithsonian previous to coming to Yale, William and I worked along with other colleagues in our team to write data set cards for a lot of our open data sets. So these can be called data statements, data cards. They're really just describing how the data were produced, who touched it, how if it's text data, was it transcribed, was it scraped from the web. What was the sort of process for these data.
It just makes it much more usable to other people who want to use it with AI tools. And then license data. This is a really big challenge and I'm sure you're all aware of is that as Bridget mentioned, the licenses are changing or the vendors are restricting more. So what can be done with license data. How do we. Enforce that.
If it's part of an agreement that Yale has signed and the library is providing access to the data, how do we and who enforces those terms about using not using AI tools on that data. And then knowing the AI tools as much as possible. It's difficult to know everything about these tools. What are the limitations of things that we know about the models that students and other researchers are using.
What do we know about the data that the models were trained on. Can we prevent hallucinations. In a research context, this is crucial. And thinking about reproducibility. How can we ensure that if you publish your work using some of these models, that other people will be able to. Or reproduce it. That's all I have.
Thank you. Thank you to all our speakers. So you've been given a bit of a landscape of what's happening in AI and then some very specific examples of how AI tools are being used in the world. So I've got a few more questions just to follow up on that, because obviously what Rebecca has also discussed is that there are still questions to answer. But a lot of headway is being made.
So thinking about the way that you have all used AI in some capacity, I'd like to ask, what do you think the unique opportunities are specifically in HSS and what have you learned from what you have done. And maybe Bridget, you could start the conversation and then Rebecca, Laura and William can add to that. We'll come up here.
And up so you can actually see me back here. But for me, the big opportunity that comes to mind in the space is thinking about lowering that barrier to entry. And that could apply in some pretty disparate examples. But just to think of a few. I know when I was a book editor in the humanities, a big topic of conversation was there was such a demand for translating work, but there wasn't. There weren't the budgets for it.
And so academics would come to us as publishers, but we didn't have the budgets for it either most of the time. And just knowing that some of these tools, these translation tools can help to make that mean they're not necessarily going to be you're not just going to put the text into the translation tool and publish it, but it can still help to bring some of those costs down. Or at least help somebody to access that content to use in their scholarship.
And so on. But then on a different end of the spectrum, if you think about product development at publishers where there still resources, a big issue, I can do things like create a synthetic data set that might be able to help your product development that is less expensive than having to get your own data house in order and structure and do those kind of big projects. So I think there's a pretty wide range of examples where that barrier to entry could be an opportunity.
I think one of the things that was most interesting for me in the last couple of months and part of it comes out of some of the work that's being done by digital scholar scholarship Fellows at the museum is that our content, they view something like the encyclopedia as a data set. And when they're looking at that content, they want to be able to pull the data out of those entries in a really easy way.
And so it made me think about our content a little bit differently, that, Yes, we need to host it in HTML and PDF so that can be found on the web and so that people can have a really nice reading experience. But then they also need to be able to get data out of it. And that's a different format. And then how do we create those files in those formats, make people know that they're there, but then also protect them.
Because while I want to make sure that our digital scholarship Fellows have the opportunity to use that really rich data, I want to make sure I know who's using that really rich data. So that was something that I had not been thinking about before we started to have these conversations around large language models. So for me, the main thing I've thought about over the last three or four years is the way in which you always hear the phrase that AI is going to take jobs and that's not really going to happen.
What we're going to see is this hybrid ecosystem in which we have what we call human in the loop or machine in the loop, where we do our jobs with AI, essentially assisting in some of the places where it excels and then doing manual validation and manual correction. And this is a good thing. And so in a lot of different projects for things like named entity recognition, you might need to train a supervised model to do this task and that's going to take maybe months, maybe weeks.
Depends on how much data you have to collect to annotate a large amount of data that can then be used to train a model. One of the things that you can do is you can use machine learning models in the loop to help the annotation process. So that you can create gold standard data sets that have been manually validated and then train a separate model to do that task more efficiently and more accurately.
But no matter what, at the end of the day, if we're working with archives that are particularly sensitive, there's always going to be a manual validation process before data goes out to the public. Institutions are not only the keepers of knowledge, but they function as kind of an authoritative source of that data. That means that once that name is on it, once it goes out, it's not going to be easy to pull that data back and it's going to have that authoritative mark on it.
So no matter what humans are going to be used in this ecosystem to constantly validate what machines are doing, but that's a good thing. It means that we can get through the backlog of collections more effectively and easily and still have the same quality if we're doing that manual validation. Yeah, agree with what everyone has said. I would add a little bit to Bridget's point about accessibility, providing being able to give image tags and to be able to meet the accessibility standards for putting content online is a great opportunity for things like special collections, which few people get to see.
But if we can use AI and machine learning to make data more available, Yes, there has to be human in the loop. And we need to potentially have custom models in order to do that accurately and well. But it's just an opportunity that we'll never have enough staff to manually catalog things in a way to be online. So that's where I see a huge opportunity.
And I think it's interesting just listening to our panelists that we're not talking about creating content using AI. What we're talking about is enhancing the content that we already have and to make it more accessible to pull out of that. The fact that humanities has data, you just if you were in the session this morning about data in the humanities, they talked about a project where they have cataloged all of the slave trade voyages across the ocean and how many people came over in each trip and when they embarked and when they disembarked.
This is all data. It's just because it's not an equation or a gel blot or a big table. It doesn't mean it's not data. And so I think we have to change the narrative. Also, when we talk about data and accessibility in the humanities and arts, because it's about your perspective. And I think that's why I think we need to have these conversations to think about this in a different way.
So a lot of what we've talked about is enhancing the content we already have and making it available to a wider audience. And I think that's a really powerful point to take away. But that also brings me on to another question that I'd like to ask our panelists, which is around ethics. And so we've talked a bit about opportunities, but what do you think are some of the challenges, especially around the ethics of using AI, especially with potentially some sensitive material that is again, it's opinions, it's people, it's about lives, it's about culture.
And how would you how would you deal with some of those ethical challenges. Do you maybe Rebecca, do you want to say anything on that first. Sure and I think it's come up a lot in what William talked about already. I think it's really just being aware of the limitations of the models you're using and what data they were trained on.
So we there's a lot of garbage in ChatGPT, so we need to be aware of is there potentially a list of terms, we would not want put on our data. We could customize a model to make sure that those terms never get assigned to our data. It's really about bringing the content experts in early, I think in identifying what they're looking for in making data more available and what are the hot button areas where ethically there will be challenges.
I can pretty much guarantee that they'll know what those are already. And so really just reverse engineering the computation to make that happen in a way that people are comfortable with. And having a disclaimer potentially things are put online, having a feedback mechanism for community members to say, this doesn't look right and to really be responsive, then I think that's the best we can do.
It's not perfect, but Yeah. So I'll talk about two different aspects of biases one explicit and one implicit. So from South Africa, we have a large collection of documents from the Truth and Reconciliation Commission which detail human rights violations essentially over the course of many decades in South Africa. One of the things that we could work into the model is we could work into the model specific labels to extract.
So we could identify perpetrators of violence, maybe the type of violence that occurred and the victims of violence. Just because you can work a label into a model doesn't mean that you should. What's the natural downstream effect of that. Generate a list that's not going to be manually validated, potentially that can identify a whole bunch of perpetrators. What if 5% of those perpetrators are wrong, which is a very likely outcome that leads to a whole bunch of not only ethical problems, but legal problems as well.
And so, like Rebecca said, working with a team of content experts and spending a lot of time coming up with a controlled vocabulary of what can and what should be worked into a model is fundamental. Another bias is more implicit, and it's not necessarily something you would notice on the surface. If you're using large language models all the time. And that's going to be the effect of low resource languages or languages that don't have as many texts digitized and publicly available.
And this is another kind of bias. This is a bias where models don't perform as well on these kinds of languages. So in South Africa, you have a collection of Bantu languages so Zosia, Zulu, et cetera. These are low resource languages because there's not a lot of digitized texts on them. So what are some ways you can overcome those biases. Well, one of the things you can do is you can go out and you can collect or digitize a lot more texts in these areas and fine tune open source large language models on this data so that those who maybe are researchers or human rights lawyers might have the same equitable access to doing tasks like named entity recognition on documents from those languages or those language groups.
So those are two different kinds of biases. Thank you. I'll see. One of the challenges that we've run into is when we were looking and working through this metadata issue with the encyclopadie idea and we still kind of are one of the other things that we considered using an API for was to create abstracts, which are another really valuable part of metadata.
But we got into it and we realized, no, I don't think that's someplace we're comfortable with. I don't think we're comfortable with content being generated through an API and then attached to our own original research. And that was just I don't think that until we saw those abstracts literally in black and white that we realized that like no, we're uncomfortable with this and that keywords are probably going to be enough for our purposes.
But I think it all links back to this common theme of even though these are really useful tools, you still need content experts who are reviewing this content. And when it comes to I think for us, when it comes to creating that content, that is still a human who is a researcher and a writer and is really skilled at doing those jobs and communicating that the research that they've done.
And so I'm now going to go to I know what you've all been waiting for is the results of the poll. So we have had some more responses since we've started. So the three questions that we posed was, do you feel Well equipped to address the challenges and opportunities of AI as it relates to your work in NHS. The majority. So we had 26 people respond and the majority said no. So that's quite definitive.
Does your organization have an AI policy. 38 people responded and the majority said Yes, but 14 people said no. And then I think one which I thought was the most interesting. Which areas of AI information do you need more support on. We had 32 people respond to this. The options were policy and publishing ethics. Six people said Yes.
Application to publishing only two people. Ethical considerations. Six people resources four people. But by and large, the one that had the most people vote for it was legal issues like copyright and licensing, where it's an obvious indication, just even from this small sample, that people are still needing support around how they can use AI, AI tools, AI in their day to day work, what the implications are of that, what they can and cannot do, and actually how do you actually implement that into your day to day work.
And hopefully today you've heard a few case studies of how it has actually been worked into projects and work that people are doing but hasn't changed. The root of the goal of what they're trying to achieve. It's just helped to get there. But it feels like there is still a lot of work to do on more practical implementation of these tools. And so that brings us on to coming towards the end of our talk but opening up to questions from all of you for anything that you've heard today.
Any other comments you may have, any thoughts. We're very keen to keep this conversation going around the humanities and social sciences, so we'd be very keen to hear more from anything that you all have been working on and would like to comment on. Great I'll come over there. Hi, Thanks so much. My name is Rebecca.
I am a colleague of. I'm going to start over. I just lost my train of thought. I'm from Wiley. I'm a senior design strategy manager and I really appreciate all of the commentary that was provided here today on the panel. What's very interesting and very timely is that OpenAI actually released ChatGPT yesterday.
So a lot of what we're talking about regarding responsible AI on campus and within on campus entities might find that as a solution. The jury is still out. As an adjunct professor, I have struggled with students using ChatGPT to generate full on research proposals and papers, so I'm interested in investigating that further. But my question is and I'm not sure if this is relevant to this audience, but I'm curious if you have interfaced with any related funding proposals or grants that have mentioned any type of cost or cost structure for integrating generative AI resources into a project.
And if you have seen that, what does that look like. Good question. Does anyone want to take that first. I can answer. It just finished a grant application that did this. OK I think I have a loud voice. I'm OK. So yeah, so one of the things that we did was OpenAI or cloud or any other service will oftentimes in the number of 1 million tokens give you the current number.
And that unless it's Google's Gemini, it won't fluctuate too much, which just went up, I think double in cost last week. So most of the time these costs are somewhat reliable. If anything, they're going to go down, especially as new models come out. So one of the big challenges is that if you're in the humanities in any age grant, let's say you start putting start drafting it, you're not going to get the grant for probably six months to a year.
Once you get the funds, maybe another three months, once you get to the stage where you're ready to start using the API. Another three months and that time maybe three new versions of GPT come out and the cost has either gone up to use the better quality one or gone down to use the ones where you started at when you started writing the grant. So what I would recommend doing what we did was we just.
Build it essentially with what we estimated to be the number of uses in the millions of tokens. This was for generating synthetic data to fine tune a large language model and we build it according to that. And we made a note that it was an estimate and that we were going to pull from the reserve funds in case we needed to use more.
Thank you. Any other questions. Hi, I may. I'm here from Emory University, where I work with humanities researchers who are publishing books. And so I'm not sure if this is a question or a train of thought. So apologies in advance, but we hear so much about humans will validate, humans will validate.
We'll just the machines will do it, the humans will validate. And I worry a little bit about whether we are underestimating how that will work and whether people might need additional training if their job is then as a validator and not as a writer. And so I wonder if any of you have thoughts about that. And then kind of connected to that. We also I mean, I've heard that in this session being mentioned. Well, we're not it's not the machine is not creating it.
We're not having the creating the content. And obviously, we can draw lines in different places. But anyone who is in the panel yesterday where Springer was like, here's our book designer, we can make whole monographs. I think that there is a we're moving in the direction of that being something that some publishers are OK with. And so I kind of wonder yeah, whether that is just going to be something you anticipate is a line.
We just need to keep discussing where it is or what it means to co-produce content, especially because the researchers I work with where they use it is in drafting. So that's the start already. And so I think that that's a question, I'd be interested to hear what you all think. Thanks so much. No, Thank you.
Very good question. So maybe I can start with this one in terms of where we are at Bloomsbury. So I think you're absolutely right. I think there is an element of training that is needed. I think and we'd be silly to think that you can just have this as part of your day to day job. So what we're doing is we've created an internal steering group where monitoring the use of ChatGPT and other tools.
We're creating guidelines for how our authors can use it, which is acceptable for us. And we're drawing a line in the sand. But you're very right. I think that line will continue to change. I don't think it's set in stone. I think as we evolve I mean, as we've mentioned, in two weeks time, there's probably going to be a newer version of something else that's come out.
So this is going to continue to evolve. And I think we have to be flexible and nimble and we have to adapt with the way that the world of research is adapting. But I still think that there is still a line. And I think each publisher and each institution has to make up their own mind about where that is drawn. I used to work at Springer Nature, so I've been in the journal world and now moving into books, and so I'm very familiar with both sides of the story.
And I think each publisher has to make up their own decision about where they're comfortable being. I think there may well be a time where we're using AI to create summaries of content, perhaps, or to create different types of content. So I'm not saying that will never happen, but I think it's a conversation that we have to keep discussing because I think it's about the ethics. I think it's about the validation.
I think it's about where your values are and what you feel. For us, original and excellence in work is first and foremost. So all the questions around authorship and copyright and whether it's the version of truth and whose truth are really important, and I don't know that we have the answers, but it's a work in progress. And I think everyone's trying to figure that out in different ways.
So I don't think there is a right or a wrong way. But I think keep having the conversations. And I think that line will continue to move. But I think also that as an industry, we need to share what we're doing and actually be a little bit more open about that as well, because that's how we will learn and develop collectively, because fundamentally we're all about the research and disseminating that research and making it available to as many people as possible.
So if we keep that at our core, then I think we can work more together. I don't know if any of you want to add something, just to speak a little bit to the second part of your question. The preprint that I referenced earlier in my slides, it did an analysis of the publisher policies on authorship. And it was clear that publishers were drawing that line.
And I think from the Committee on publishing ethics. The difference is like, yes, you can generate content, but GPT or AI tools can't take responsibility for that content. And so that's why we're seeing those policies. So even though there's these kind of experiments that are getting buzz like the book that was written by AI, by Springer Nature, as far as those policies, I don't see any indication that that's necessarily going to change.
But my colleagues might have some thoughts on the first part of your question. One of the things when we started to think about how people are going to check these keywords because that's going to now need to be a part of our process over, I think the next six months for thousands of entries is that it's a resource issue. Like we've now done this and we've gotten further than we would have gotten without this tool.
But now we've created more work that needs to be done and we now need to get it done very quickly in order to get this product launched. And I think that's something what we're starting to grapple with a little bit. And one of the things to your second question, one of the things that's kind of resonating from in my mind from that conversation yesterday and with an AI generated book is what is the role of the researcher.
Is the role of the researcher to do the research or is the role of the researcher to do the research and be able to communicate that research. Because when you have an AI suddenly step in to do the writing and then that removes part of that role of the researcher. And so it reshapes that and I think it reshapes it in a way that is I mean, I can speak for where I'm coming from.
It reshapes it in a way that, for humanities I'm not really comfortable. That's part of this process. And that's what makes it unique and that's what makes it a really wonderful product. And that's also how you learn more about the research you've done is through that writing process. So you start to remove that and I worry it starts to unravel a larger part of the academic ecosystem.
Yeah, and sorry, I'll take it one more step further. I think even more so for the Humanities because especially if you went to Karen Wolfe's the talk they did yesterday about the crisis in the humanities and what are the really big skill sets that you come out of college, undergraduate or graduate or PhD from a humanities major. It is writing clearly, it is doing good research. And those are skills that you use to then later further your career.
And so if we start to erode the value of those, it starts to really impact that in a much larger way. Yeah, I totally agree. Anyone else. I was going to say, there's really two things that lead to a better outcome with human in the loop. The first one is going to be a well defined schema and that's going to look like the things that you want to extract.
So the labels a description that's very clearly defined so that they can reference back to it and also examples that they can go to. And the second thing that you include once you've seen a lot of different annotators do things is adding an option for I don't know or this needs to be this needs to be discussed just by giving annotators that out, you're going to remove a lot of really bad annotations almost immediately.
And make it very clear to your annotators if you're not absolutely certain, hit that button, it's OK, and then you can convene as a group and discuss those more challenging issues. And those are going to be the ones that help you rethink the schema and rethink how you're going to structure the project. And especially earlier on when you're doing the annotating with the human in the loop, having regular meetings to discuss what you're finding, what's not really aligning and adjusting early on, so you don't have to go back and redo everything.
Any other questions. Yes hey. Hi my name is Avi. So when we talk about the LMS or tools for publishing, the one question that comes to me is, so all these elements are trained to September 2022 data or not the updated data. So when we use this LLMs for the research integrity or peer review management, if you want to publish in this domain or stem domain.
So do you think. Should we check the integrity of its outcome because as it's not trained on updated data. So because that could be a problem for a peer review management or a research integrity. So any thoughts on that. Yeah, I can. That's a huge problem.
And I think that's not at all solved how to deal with that. I think particularly a lot of students are using ChatGPT to generate code and it works pretty well. If you're a new coder and you just want to make a plot of something great, but if you're trying to follow along, like maybe you're doing it in two windows and also looking at Stack Overflow and that you might not get the same answer because ChatGPT is out of date.
And that is a. That a. It challenging. So you really have to know that going in that, OK, I can maybe use this as a template, but I'm going to have to modify it if I'm actually going to use it for a research product. And that I think is really important to convey to New users because you can't really take it at face value.
I mean, you definitely can take it at face value what you get out of it. And there's two ways to really overcome limited knowledge. One is going to be hooking it up to a knowledge base in some capacity. Oftentimes, this is going to be a vector database and you can use what's called retrieval augmented generation or RAG, which is going to basically perform semantic search first based on some kind of query, go out and find the relevant documents or the primary sources, and then use those as what we call a context to improve and give the knowledge necessary to answer whatever that question might be.
This is really good, not only with outdated material, but with material that's really highly domain specific. So maybe a collection of oral history testimonies or oral testimonies from the Holocaust that weren't worked into the model that have individuals who only appear in this one document and all the collection of public documents. So that's one that's useful. Another thing, another approach is I lost my train of thought.
There is a second one. I promise you, when it comes to me, I'll say it. Sorry, I remembered. The other option is to fine tune an open source language model on your specific domain texts. This is really useful if you're working with a very specific language or a highly specialized collection of documents, like something from maybe the medical industry or a particular area of historical research, like maybe early medieval Europe.
We're at time now unless there was any other questions or comments. Yeah, sure. Thank you so much. This is for Laura. I'm Nia for Brown University library. And I was just wondering, have you had discussions or are you already planning to indicate to your users or readers in some way that the keywords were generated with AI.
Yeah, that is part of our conversation. And I think one of the things that makes the way that we're using it a little different is that if we were putting those keywords on our site, we could very clearly say these keywords were generated using AI and checked by a researcher. But because they're going into our metadata and so they aren't available to the standard user, they're going out as this upload to these discovery partners.
I think we're going to need to work on what does that look like and what's necessary for us to put in there as a caveat. But I'm also happy to continue the conversation as well. So I'm going to close the session. Thank you again very much, everyone, for coming. Thank you to our panelists. And if anyone does want to continue the conversations, we're really happy to speak after the session. Thank you so much.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode