Name:
Open access and analytics
Description:
Open access and analytics
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/4f2cf3f3-3721-4c5a-b212-113c5ada7a2a/videoscrubberimages/Scrubber_1.jpg?sv=2019-02-02&sr=c&sig=OLKwDexpAvyT6E%2FbhNPQyJ1gC7bRl3uQT0WyfgXBNZE%3D&st=2024-10-16T01%3A56%3A26Z&se=2024-10-16T06%3A01%3A26Z&sp=r
Duration:
T00H34M19S
Embed URL:
https://stream.cadmore.media/player/4f2cf3f3-3721-4c5a-b212-113c5ada7a2a
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/4f2cf3f3-3721-4c5a-b212-113c5ada7a2a/43 - Open access and analytics-HD 1080p.mov?sv=2019-02-02&sr=c&sig=voMtrlWuqN3GQt3Zf43NoDCw9juQ8f%2BJ5%2B17tMYXWTU%3D&st=2024-10-16T01%3A56%3A26Z&se=2024-10-16T04%3A01%3A26Z&sp=r
Upload Date:
2023-02-13T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
[MUSIC PLAYING]
MICHAEL HABIB: Hello. I'm your moderator Michael Habib, a product director with Clarivate. COUNTER metrics were originally designed for reporting usage of subscription journal content to institutional subscribers. However, the growth of open access has introduced new business models, and with them, establish new relationships between academic institutions, open access publishers, and additional stakeholders, such as funders.
MICHAEL HABIB: With those changes, COUNTER in its original design is no longer sufficient for all stakeholder reporting needs. Our panel today will explore what the current limitations and challenges are with usage reporting for open access content. First Lorraine in COUNTER, we'll discuss the current status quo in COUNTER for open access with a focus on open access books. Next we'll learn from Clark & Esposito and Brian from the Book Industry Standard Group.
MICHAEL HABIB: We'll discuss COUNTER and usage reporting for open access books in more detail. Lastly, Sara from PLOS and Tim from LibLynx will dig into a case study around COUNTER and analytics for open access journals. We will then adjourn into breakout sessions, looking at opportunities to fill gaps in the current system. Each session will have a focus on either books or journals. With that, I will hand it off to Lorraine.
MICHAEL HABIB: Thank you.
LORRAINE ESTELLE: OK, I'll start. Hello, and welcome everybody. I am Lorraine Estelle from COUNTER, and I'm going to be talking to you today about the trouble with books. The trouble with books is that the same book can be delivered from multiple platforms. And those platforms differ in how they deliver books. Some deliver books as whole files-- that is to say, the whole book is a single PDF.
LORRAINE ESTELLE: Others deliver books cut up into individual chapters. Some platforms deliver individual chapters, but bundled together in a ZIP file. And of course, there are various other methods. And this means that it was challenging to compare usage of the same book on a different platform and impossible to aggregate usage. Next slide, please.
LORRAINE ESTELLE: And this is why COUNTER release 5 developed three new metrics for book usage-- Total_Requests, Unique_Item_Requests, and Unique_Title_Requests. And these metrics are designed to enable comparison and aggregation at the title level. Now, you may be wondering why I have the cute cat. The cat is going to help me explain how these metrics work in action.
LORRAINE ESTELLE: Next slide, please. We're going to look at two scenarios to see how the metrics work. In the first scenario, we have platform Abietta. And Freddie downloaded two chapters from The Book of Cats. Now, his counts are Total_Requests, 2; Unique_Item_Requests, 2.
LORRAINE ESTELLE: Each of those chapters is a unique item, and unique title requests is 1. And that's because both those chapters came from the same book title, The Book of Cats. Now, over on platform Liberdad, Sophie also downloaded The Book of Cats. But on Liberdad, books are delivered as single files. So the counts for Sophie's actions are, Total_Requests, 1; Unique_Item_Requests, 1; and Unique_Title_Requests, 1.
LORRAINE ESTELLE: So you can see that the Unique_Title_Request metric really enables us to compare usage across different platforms at the title level. Next slide, please. But of course, we're still interested in what people use. And that's where the COUNTER release 5 attributes come into play. Attributes tell us about what was used.
LORRAINE ESTELLE: And we use the term Section_Types for these attributes. And you can see here is the list. A Section_Type can be a book-- that's a book delivered as a single file-- a chapter, other, and section. So that really gives us a really good idea of what was used across different platforms. I'm just going to finish briefly by telling you a little bit about the work that COUNTER is doing in the area of open access.
LORRAINE ESTELLE: So next slide, please. At COUNTER, we have recently pulled together an open access advisory board. And that board is really looking at the horizon and informing the strategic development of COUNTER and the code of practice in response to developments in open access. Jill Emery is a member of COUNTER, and she is chair-- sorry, and she is on our executive committee.
LORRAINE ESTELLE: And she is also on the advisory board of the open access e-book usage project. So she's making sure we're linked to that project and feeding back. Jill also chairs another come COUNTER subgroup, the OA/unpaywalled subgroup. And that subgroup is doing the heavy lifting. And they're looking at the how. So they're looking at how we can enable open access reporting.
LORRAINE ESTELLE: So a lot of work going on. Please watch this space. We will be communicating the work we're doing in this area and keeping you all informed. Now I'm going to hand over to the next speaker, Laura Ricci. So Laura, over to you.
LAURA RICCI: Thank you, Lorraine. I'm excited to be part of this session. And again, my name is Laura Ricci, and I'm a consultant at Clarke & Esposito. Next slide, please. The Clark & Esposito was recently brought in by the participants of the Exploring Open Access Book Usage project, which Lorraine just mentioned. And they asked us to analyze and map the supply chain for OA books.
LAURA RICCI: And the goal of this was to understand how OA monographs travel through the world, whether it's on one platform or many, and how usage of those e-books can be collected and communicated back upstream so that the content creators, funders, and other stakeholders in their creation can understand the impact their work has had. And what I'm going to focus on is going to add to what Lorraine has described and talk about the gaps we found in that whole process of information exchange and what opportunities there are to improve.
LAURA RICCI: And these are opportunities not really for any one provider, but for the whole community. And so it's perfect, I think, to set us up for discussion about ways the broader community might work together. Next slide, please. So the first thing we learned is that for the supply chain, even when serving open access books, the incentives are still aligned with paid access.
LAURA RICCI: A lot of the participants in the supply chain grew out of the traditional books supply chain, which is really organized around the central concept of sales. So distributing a free book doesn't provide them with the same value. At the same time, there's not really a pure open access supply chain you can spin up because even open access books can retain a sales component.
LAURA RICCI: For example, there can be an open access book with a print version. That's pretty obviously going to be for sale. So because sales isn't really a meaningful concept for open access books, these providers which straddle the open access and paid access book world need to decide on a new incentive to get those books moving through the system and build the appropriate infrastructure, and by that I mean standards, around those incentives.
LAURA RICCI: Next slide. So the next challenge we found, of course, is that when trying to develop standards for open access books, we found a lot of borrowing from the journals model. And this is grade up to a point because you want to learn from experience, but journals are really fundamentally different from books. So for example, journals have evolved to support hybrid titles with both OA and subscription articles pretty well.
LAURA RICCI: But most booksellers really only sell whole books. So distribution standards are built around whole books. A hybrid book, which costs money but has no OA chapters, pretty much just a headache at this point. And even something like the DOI, which works great at identifying journal articles, gets messier for books. So sometimes chapters have their own DOI, but most often books have a full DOI, except sometimes books are on multiple platforms.
LAURA RICCI: And so book has multiple DOIs, like uh, that's not great, but it happens. And there's this perception that books are somehow behind journals in adoption of some of these standards. But really, it seems like the more productive path is to make sure the standards reflect how books are actually made and sold so that we can better talk about exactly how the book is open access. Next slide.
LAURA RICCI: So of course, of books passed, open access is not always going to be the same either. And that means the metadata is always moving. And we saw a big gap where the distribution process can sometimes struggle to keep up with the book's away status. So let's say a book is out for sale already and then, at some point, that book becomes open access. It's been flipped. The book publisher can try to use the same ISBN and overwrite pre-existing metadata, but if there's a distributor somewhere that rejects zero-dollar price points, which is a common thing, then the update might fail, and a paid access version of that book is going to persist in the marketplace.
LAURA RICCI: And of course, that's going to influence who's able to see and use it. Now, sometimes you see a paid access version of an OA book, and that doesn't mean that anything's necessarily wrong. Sometimes publishers are OK with having OA book alongside a paid access version. There's really no way of knowing from just looking at it that something's wrong. So the only way to solve this is for the whole supply chain to properly recognize both OA and paid access metadata elements all the way through so that when a book is made OA, everyone agrees, and the book is available on that basis.
LAURA RICCI: Next slide. Now, getting people to adhere to standards is easier said than done. Of course, seeing as this is an ISO PLOS presentation, I'm sure this is news to exactly no one here. But what adds to this challenge is what Lorraine alluded to, about the difficulty of aggregating usage of books across multiple platforms, especially when those books are away and don't require authentication to access.
LAURA RICCI: The standard is really built to communicate to librarians, even though, in this case, it's the publishers, authors, funders who care most about how an OA book is being used. So COUNTER does guide a lot of reporting back to publishers and authors and funders, but there's definitely room to keep working and develop best practices that are addressing the specific use cases of OA, and in particular, OA books. Next slide.
LAURA RICCI: And what's most challenging about OA books? Well, you guessed it-- it's the supply chain because unlike journals, books distribution is about getting a title through as many channels and platforms as possible. So that means there's no one single version of record and no one single source of usage. So for a publisher who wants to prove their OA books are being used, this creates a whole maze of different reports and metrics and measurements, and it's really hard to define success.
LAURA RICCI: But we're having this conversation today because this is a problem that's been recognized. And a whole point of identifying these gaps is to get people working together to solve them so that when a book is supposed to be OA, it is available OA. And when an OA book is used, their creators know about it. It might be hard to get all of this perfect, but this is how providers can start to align their incentives.
LAURA RICCI: So this is where I will turn it over to Brian O'Leary of the book industry study group, who will talk more about these efforts. But first, I will add that these findings were all included in our written report for the project which is currently open for public comment. So you can be a part of this discussion, and I'm sure Brian will have more to say about this too. But thank you for your time.
LAURA RICCI: I'll turn it over, and I look forward to furthering the conversation online and in the breakout rooms.
BRIAN O'LEARY: Thank you, Laura. I'd like to start with a little bit of an introduction to the project that kind of put Laura, Michael, and I together, which is the open access e-book usage project to develop a data trust to essentially create a better understanding of usage of open access monographs. A data trust itself is an independent intermediary among industry stakeholders. It typically would compile and analyze data on behalf of the members of the data trust itself.
BRIAN O'LEARY: There are normally agreements-- and we're working out the specifics as part of this project-- agreements to share data among the members of the trust. But they have to operate in a way in which the contributors, who could include everybody from authors and publishers through to analytics providers as well as distributors and others, to respect their confidentiality and privacy so that the data that's shared is shared in a way that maintains the wishes of all the members of the trust.
BRIAN O'LEARY: The thing that's a real value is that it supports benchmarking and provides the opportunity to understand data, more widely than would be the case for individual members or for any one data source. In this sense, the notion is that you would write once, meaning that you would gather all the information and then share it in a consistent way across many different uses. Now, if we can go to the next couple of slides, we'll include a chart as well as a link.
BRIAN O'LEARY: If you look down at the bottom, that's the specific comment draft for the open access book supply chain, this map. It's a little bit of eye candy here, but moving from left to right, you have an opportunity to look at the rules for content creators and funders as well as distributors, content platforms, sales channels, catalogs and indices, and the end users. The thing that you see within this is obviously a demonstration of the complexity that Laura outlined just a moment ago.
BRIAN O'LEARY: It's one of the reasons that we chose to conduct this research because we feel that understanding global usage and the detailed impacts for open access e-books is both complicated. And to be able to come up with a solution, we have to embrace that complexity to a certain degree. We also want to make sure that all of the stakeholders that are involved in this, even at the highest level, is a pretty good set of buckets that have the opportunity to tell their stories of impact that makes sense across the entire supply chain.
BRIAN O'LEARY: So if you go to the slide that follows, this visual indicates that there are multiple flows and sources of usage and engagement data. The same set of actors that were outlined in the previous slide appear here, but this diagram shows the variety of ways in which information about usage is both generated and captured. You can see, automatically, that there's lots of moving parts, and as Laura indicated, not as much consistency across various use cases.
BRIAN O'LEARY: Generating analytics, data-driven reports, and dashboards, when your data is sourced in so many different locations, winds up being complicated, time-consuming, and challenging without necessarily offering the right depth and breadth of data science expertise. And it also raises questions about data ethics, scholarly freedom, and privacy when you're conducting this collection analysis at scale.
BRIAN O'LEARY: We see, in the next slide, three big challenges to resolve. The first is creating this data trust platform. It's got to be able to deal with issues of data interoperability, a relatively straightforward way of onboarding, and an important and critical way of maintaining trust, both in the creation and in the day-to-day maintenance of the trust. The second is, how do you process data to create a multi-stakeholder framework?
BRIAN O'LEARY: The significant work that Lorraine explained right at the outset has taken several years and several versions to create for COUNTER to try and create comparability across multiple measures of different types. It's also particularly challenging. And then the third piece is visualizing the data that we've processed, making it representative across wide-- what are often widely varying use cases, meaning that one publisher may want to be looking at its data in a specific way.
BRIAN O'LEARY: Another may want to layer in things that we track but don't necessarily present. That kind of thing is one of the things that we're trying to think through. And the quote at the bottom here is drawn from the white paper that kind of led to this project. But it points out at the beginning-- at the end, rather-- that the solution that we're trying to develop requires the development of a trusted framework for coordination across all relevant stakeholders.
BRIAN O'LEARY: And that's why it's a two-year project. I can go to the last slide, which is a little bit of a preview. We have three questions for our breakout group. These will be repeated at the end. The first of which is-- we're just asking out loud-- what measures of open access book usage are working now, and how might they be improved from your perspective?
BRIAN O'LEARY: Do the supply chain recommendations that Laura outlined address key concerns of the things we're missing? And then the last is, can a data trust potentially improve how we measure open access book usage? We'll come back to this at the end, but that's kind of what we're thinking about. And with that, I'll stop and introduce Sara Rouhi from PLOS to talk about next-generation open access analytics.
SARA ROUHI: Thank you so much, Brian. Good afternoon, or morning, wherever today's webinar finds you. My name is Sara Rouhi. I'm the director of strategic partnerships at PLOS, and my colleague Tim Lloyd, the CEO at LibLynx and I will be doing a presentation of the current case study we're working on around next-generation OA analytics. Tim is going to be doing the bulk of the presentation, but a brief overview to provide context of our work really relates around the new challenges or the new opportunities publishers, libraries are facing now around assessing the impact of native publishing agreements, so pure publish agreements, like the ones PLOS is now negotiating, or read-and-publish agreements, where a large component of the content that's being paid for is not the content to read, but the ability to support the cost of publishing.
SARA ROUHI: And the result of that, obviously, is that the content is open access. And so the historic metrics that COUNTER provide that are incredibly useful for assessing paywalled articles and content no longer really apply when open access content can be available anywhere at any time across the globe, with no need for IP authentication, which is obviously the main way that COUNTER is able to look at usage statistics.
SARA ROUHI: So our partnership was really looking at, what are the next generation of usage statistics going to look like? And how will they inform how libraries, and other stakeholders look at the impact of an agreement with PLOS or any other OA publisher in determining, in future, do we renew these agreements? Or at a more-- kind of in more general level, what can these metrics tell us about the impact of this research?
SARA ROUHI: So this has been a partnership that we're undertaking currently with LibLynx. They've been a fantastic partner. And with that, I'll turn it over to Tim, thanks.
TIM LLOYD: Hey, great. Thank you very much, Sarah. So in this presentation, I'm going to talk a bit about the current state of analytics in open access, why we think innovation is needed for open access content. I'll review some of the use cases that we're seeing for open access analytics that we're encountering as we talk with PLOS and their stakeholders.
TIM LLOYD: And I'll show you some of the reports we are working on. So I think the first thing to underline is that it feels very much like we're in an era of experimentation right now, which is probably similar to the way paywalled analytics was before COUNTER appeared. We're seeing a lot of innovation in business models, developing in addition to the traditional ways of funding gold open access to article processing charges, so subscribe to open or PLOS' community action publishing model.
TIM LLOYD: In turn, this is generating a variety of different goals for usage analytics. So they can range from simply counting usage of a model to identifying organizations that have accessed that open access content, all the way through to pretty granular analysis of topics or subjects to support editorial development. Organizations are also using a variety of different metrics to support this activity.
TIM LLOYD: So those relying on generic analytical tools like Google Analytics or Adobe Analytics will be getting very generic measures, like sessions, paychecks, and downloads. Some are used in COUNTER metrics. And some are crofting home-brew solutions. There's a proliferation of stakeholders interested in analytics far beyond the library. And I'll come to that in a little bit.
TIM LLOYD: And there's a lot of money at stake. Much of that money is currently sunk into traditional paywalled models, and open access analytics are a crucial tool in informing negotiations on all sides. And another large chunk is sitting with funders who influence where the research they fund gets published. So what about COUNTER Reports?
TIM LLOYD: And these sound like these should be able to support this. Well, they don't actually, but it's because they're already engineered for a very specific use case to help a librarian understand and compare usage of paid content to make informed decisions on acquiring content to meet their goals. So this use case, rightly, focuses on the aggregate numbers. It prioritizes scalability over usability because in most situations, these numbers are being aggregated further and analyzed using software.
TIM LLOYD: Might be reporting using [INAUDIBLE].. But that doesn't mean the COUNTER doesn't have a role to play, and far from it because, as the quote says at the bottom, "Metrics are the gold nugget at the heart of COUNTER." Think of metrics as a defined set of valuable LEGO bricks that we can recombine to create new and valuable analytics reporting.
TIM LLOYD: So what's this project we're working on? So it's a partnership between PLOS and LibLynx. And there are three goals we set ourselves. So the first one is to understand stakeholder needs from open access analytics. So in the process of soliciting feedback from a range of internal and external stakeholders across the PLOS community, and some of that feedback's already reflected in the analysis I'll be showing you shortly, and the process is ongoing.
TIM LLOYD: The second goal is to provide COUNTER Reports to meet the community's immediate needs to better understand the impact of open access content published by PLOS. And the third goal is to develop these next generation analytics that can meet more diverse use cases that we're seeing in the open access environment. And I'll be talking a bit more about that as well. So let's start off by looking at some of the roles that we're seeing for using open access analytics.
TIM LLOYD: And one way into this might be to start by looking at the stakeholders in the process. So we've got research institutions that are typically generating the research that's published. We've got publishers that publish open access content. This could be the institution itself or a specialist organization like PLOS. We've got the authors for this research. We've got the community that's interested in reading the research.
TIM LLOYD: We've got funders that pay for the research to be published. These could be institutional budgets or separate entities like the Wellcome Trust or the Gates Foundation. And we've got a number of intermediaries, performing a variety of functions that support the publishing workflow. So in that fairly complex landscape, what sort of roles use open access analytics? So the most immediately obvious one is the traditional library role, which sits at the core of the COUNTER reporting use case.
TIM LLOYD: So libraries are already tracking usage of open access content as part of license collections. Those that also play a role in pure open access publishing will be equally interested in monitoring that usage. There's also a role for libraries to explore understanding usage across both pure open access and paywalled content. We've also got institutional roles that sit outside the library and the focus on research management, such as the senior research officer role.
TIM LLOYD: And if you haven't already read Roger Schoenfeld's excellent December Scholarly Kitchen post on that particular role, I would thoroughly recommend it. So in that sort of scenario, this role is going to be more interested in understanding how usage of open access content ties into institutional research priorities. And that's got some interesting implications. On the publisher side, we've got development rules that need to understand which organizations are getting value from open access content in order to identify potential future sources of funding.
TIM LLOYD: And depending on whether the publisher is not for profit or commercial, the nature of that development can vary. But they're all after very similar information. We've also got editorial rules that want to understand the subjects and topics that are engaging the community. We've got authors, who want to understand the impact of the research they publish.
TIM LLOYD: And in some cases, this information can influence their choice of open access publisher. Similarly, funders, they want to understand the impact of the research they funded. Did it reach the community they were targeting? Or perhaps it also got engagement from new communities they previously weren't aware of. And then these various intermediaries also have interests and stakes here.
TIM LLOYD: So whether you're a service provider like, say, Knowledge Unlatched or you're a consortia or similar organization, like [INAUDIBLE] in the UK, or you're a distributor-- perhaps you're distributing e-books like JSTOR. And then last but not least, the community itself-- this is particularly important for publicly funded organizations, like, say, state universities who want evidence that the open access content they're funding and/or publishing is reaching the broader community they serve.
TIM LLOYD: We're also seeing demand for a broader range of metrics. So in addition to the traditional COUNTER metrics of searches, investigations, and requests, here's some other examples. So one of the metrics around the audience-- now, what type of audience is engaging with open access content? Is it identifiable organizations or simply unknown geographically?
TIM LLOYD: Where is the usage coming from? Maybe which categories of organization. Is it government bodies or higher education or corporates? And what are the specialties those organizations focus on, such as biomedical versus cancer research. Oops. Another is, what content is of interest to that audience? So this requires metadata on the subject topic of open access content, could be really interesting for roles, again, like senior research officers or authors, funders, editors involved in shaping future content priorities, for whom very detailed article-level information is actually less interesting than extracting it upwards to understand research areas.
TIM LLOYD: Another example would be funding. It will be interesting to see the impact that funders have on open access content or maybe portfolio analysis on funding, in terms of the communities impacted by making content open. And then there's a sort of grab bah of other attributes of reporting that we're seeing in relation to open access use cases.
TIM LLOYD: So one is the granularity of the reporting. For example, which organizations access what open access content when and from where? In comparison to more aggregated traditional COUNTER-reporting style. So in use cases where the usability is more important than the scalability, it's not hard to imagine that visually rich layouts that made for easy consumption would be of interest.
TIM LLOYD: There's immediacy-- so again, with COUNTER Reports, they're monthly. There could be other calendar-based formats. But there's lots of use cases where having more real-time usage reporting will be of interest to people. And then as to the format-- now, as the number of use cases rises, so does the ways in which users wants to consume that analytics.
TIM LLOYD: So it might be over the web. It might be in a PDF format that's easy to distribute. It might be automated through an API, which is bulk export so that people can do their own detailed analysis. So let's look a little bit about the reports that we're actually generating. So this is going back to goal two of our project, and that was to deliver COUNTER Reports.
TIM LLOYD: And so for several months, we've been generating-- processing data from PLOS' open access content. We've been matching incoming IP addresses against the IP registry in order to identify the organizational sources of access where possible. And we've identified over 16,000 organizations in a couple of months. And then we're using those identifications to affiliate-- associate usage with those entities.
TIM LLOYD: So the example you're seeing here is a item master report. So if you're familiar with COUNTER Reports, it looks the same as it would otherwise. The only difference is that, in this case, we've identified this usage by identifying that registered Stanford University IP addresses were associated with this access. And so here's an example where you can see a list of 783 PLOS open access articles that were investigated from registered Stanford University IP addresses in December 2020.
TIM LLOYD: And these COUNTER reports will be going live to PLOS customers in the first half of February. We've also been working on some prototypes, the more interactive analytics that could meet these broader use cases. And so the idea is that these would provide access to real-time, granular data with interfaces that would facilitate inquiry and support a broader range of metrics and related metadata.
TIM LLOYD: So this first example is a report enabling users to pick an organization associated with the usage and then drill down to the articles and authors that users from that organization engaged with. In this case, we've got the World Bank. Example two, flip that around, pick an article, and drill down to the organizations that access that particular article. Example three shows the geographical spread of access for a particular article.
TIM LLOYD: And in all of these cases, you can then interrogate the analytics with multiple dimensions, such as [INAUDIBLE] or the usage of a particular article from organizations in Spain that are governmental in nature. So there's a lot of interesting inquiry that can be done in the state once you start layering more metadata on top of it. And then we've got, similarly, some questions that we'd love to explore with you as part of the breakout sessions.
TIM LLOYD: So the first one is, "How has your thinking about usage data and other volumetrics changed, given the accelerated push for open access in the last 18 months?" Because we're certainly seeing that there's an acceleration in interest to conversation around this area. The second one is, "Where can these next generation metrics support your thinking about 'impact' from the researcher/faculty or library/collections perspective?" And third, how can these new metrics work alongside existing COUNTER metrics to present a broader understanding of the value of open access publishing?
TIM LLOYD: And if you have feedback and thoughts, please share it in the breakout sessions or contact Sara and I using the emails there. Thank you very much, and we're looking forward to talking to you now in the breakouts. Bye. [MUSIC PLAYING]