Cadmore media player playing video Measure with My Heart: Outcomes over Outputs
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
Measure with My Heart: Outcomes over Outputs
Description:
Measure with My Heart: Outcomes over Outputs
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/93b85a1a-417b-4100-ae0c-ebd1db424d76/videoscrubberimages/Scrubber_1.jpg
Duration:
T00H47M51S
Embed URL:
https://stream.cadmore.media/player/93b85a1a-417b-4100-ae0c-ebd1db424d76
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/93b85a1a-417b-4100-ae0c-ebd1db424d76/SSP2025 5-29 1045 - Session 1E.mp4?sv=2019-02-02&sr=c&sig=eMvzfe%2BgJNyt9nUUDEJV77pbKPS8QtSnyQQUzckcMSM%3D&st=2025-12-05T20%3A56%3A02Z&se=2025-12-05T23%3A01%3A02Z&sp=r
Upload Date:
2025-08-14T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
Hi, everyone. Thank you so much for being here today. We're so delighted to have you in the room with us, and thanks also for loving us through it. As we got you seated at these tables, we'd like to welcome you to I measure with my heart outcomes over outputs also during this session. Not on the slide and we probably should have put her on it as Susan Wilner.
And she is the shepherd of the session. And if you need any help with an extra pen or paper or anything like that. And she's been wonderful. Thank you Susan. Also, one of our co-panelists is Gabe herb, an amazing friend of Lete and mine, and also one of the most brilliant people we know. Unfortunately, he was unable to make it here today, but he was a full on participant in the shenanigans to put this session together.
And I have to say, we're going to miss our planning calls for this. My name is Jennifer rogala. I'm an associate director of publishing at Wolters Kluwer health, and this, I will just start by saying that this conversation was born from me telling Gabe and then Lete that I like to measure with my heart. Unfortunately, though, my employer doesn't agree and does like to have some measurable items coming out of my work, so I'll just leave it there and let Lete introduce herself.
Cool thank you. And I'm Lete Conrad, product experience architect at leblancs. And you are absolutely in for a treat. As you heard in the keynote yesterday, Doctor Schiffman mentioned the need to switch our minds from thinking about outputs, deliverables to outcomes. The bigger picture, and that's where you'll find your heart is really looking at that bigger picture.
So with that, let me launch right into a couple of introductory comments. So our code of conduct, I'm sure you've heard in other sessions really important to remember. We're committed to diversity, equity, and providing a safe, inclusive, and productive meeting environment. So avoid harassment. Avoid discrimination.
Avoid hostility, avoid anti-competitive. Comments just be kind. Be kind. Let's do that. And these are the values that SSP. As you've heard over and over, I'm sure community adaptability integrity and inclusivity. So what we're going to do today is think about some abstract concepts and think about some hypotheticals to apply to those concepts.
But every single one of us in this room has some need to think about how we measure what we do. So I'm hoping that you've brought your thinking caps today. I'm hoping you're up for a thought experiment. The table that you're sitting at is going to be a group that we will put together in just a few minutes, but let me start with a couple of basics.
So what do we mean when we say output. Basically dictionary definition for merriam-webster is something produced, an act, or a process or instance of producing. It's a deliverable. It's the thing I think most of us are really focused on every day in our jobs. I've got to tick that box. I've got to complete that task.
I've got to deliver the thing. The outcome is what is affected by our outputs. The outcome is actually a change in human behavior that drives business results. Or another way of putting it is it's a change in how our users, our customers, our stakeholders, our colleagues experience what we're doing in that it delivers what we're aiming for.
So we're looking to instead deliver the thing right, tick all the boxes, complete all our tasks. Actually, we need to go a level up and think about changing the behavior and the experience that we're delivering, the behavior of our customers and our users, and the experience that we're facilitating. Now, this is from the Kellogg model, which if any of you have worked in nonprofits, you may have encountered this. This was developed as a way to help organizations think about funneling all the various resources, fundraising materials, donations, et cetera into the things that will help them achieve their goals.
So nonprofits are often very focused on big picture things. Reducing a spread of a disease. Increasing access to clean water. But getting to that impact requires a whole step of processes, the inputs alone. Our donations from fundraising that isn't going to get better access to clean water, that isn't going to necessarily reduce the spread of disease.
The activities that are made possible by those inputs, those activities alone are not going to deliver on what we hope to achieve. But the inputs and the activities together can help us deliver the outputs. The outputs are those things that we're ticking off the task or completing the things that we're delivering. And all of those together then produce the outcomes, the change that we're looking to make in our communities, the change in behavior, the change in the way we talk about things, the change in experiences, those things together add up to impact.
And I know this is really broad at the moment, but we're going to make things a little more concrete. So we're going to start to look out for examples. The rest of our time together. We're going to be talking about examples of these things. So instead of trying to define it way up here in the abstract ether, we're going to make some concrete examples from all of this.
So here this is Gabe. Imagine I'm Gabe and I am now going to show our first example. So this comes to you from Gabe. This is an example outside of our industry. This is a real world non-scholarly example, Boeing's 737 MAX aircraft. The context is in the late 1910s, Boeing aimed to compete with Airbus fuel efficient A320neo by developing the 373 max, the 737 Max.
The company's primary goal was to deliver this new aircraft quickly to market. The urgency led to an emphasis on production schedules and delivery numbers. They led to a focus on outputs to meet those market demands and maintain competitiveness. Boeing succeeded in its output goal. It delivered the 737 Max within the desired timeline. So what's wrong.
This actually came at a huge cost. The focus on outputs came at the expense of critical safety considerations, which were essentially the outcomes they were hoping to deliver. The aircraft was equipped with maneuvering Characteristics Augmentation system, a new software feature that was intended to enhance flight stability. Adequate pilot training and comprehensive understanding of the software were not prioritized.
So the consequences. Two fatal crashes of the 737 Max occurred in 2018 and 2019, resulting in the loss of 346 lives. Investigations revealed that the software played a significant role in both accidents. The aftermath led to a global grounding of the 736 737 Max fleet, substantial financial losses for Boeing, and a tarnished reputation. These events the critical importance of prioritizing outcomes such as passenger safety and trust over mere outputs of production speed and delivery numbers.
So there's example number one. And then I'm going to share a scholarly publishing example. I mentioned that I work at Wolters Kluwer health. I've been there for exactly one year. I started last May. I oversee a team of publications professionals who are amazing. One of them is in the room right here, Caitlin, and we report on medical news in various specialties.
So we're going to talk about emergency medicine news today, a year ago when I started, we knew that emergency medicine news was the most read print publication in emergency medicine. We knew that from our Canter scores, we also knew, though from other surveys and from those same Canter surveys that emergency medicine physicians spent the majority of their time not with publications, not where we expected them to be.
But on Reddit. They were going to Reddit for all of their information, 40% to 50% of them. And that was their first resource. And that made me as a patient and just a human being, clutch my pearls a little bit. That was frightening. So knowing that we set out to figure out how can we build a vibe.
But again, I do work for a commercial publisher and vibes don't really drive what we do, so how could we build the relevance of this publication, make it be more than just a well read print publication, but make it be a community. Make it be. I truly believe every publication is a fingerprint. Unusual different from any other publication that's out there. And once you acknowledge and embrace that can make that publication more than that can make it into a community.
So we wanted to create that vibe, but we need outputs to deliver. Again, I can't just say, oh, it feels really good and we're vibing out over here. We had to be able to increase the relevance. We had to increase the reach of the publication. We had to increase how many people were receiving it, reading it, engaging with it. We also had to increase the outputs that we delivered as far as better, more podcasts, webinars, things like that to get the community involved.
But what have the outcomes been. One year later. The vibe is there. I can tell you our revenue is up, our relevance numbers are up. So my employer is happy. So that's good. So for anyone, some other Wolters Kluwer friends over here.
So we're doing that. But the outcomes are really good too. That vibe is happening. We've had in-person events where we've had people spilling out onto the street, and they're all based on emergency medicine news. We have multimedia events that are happening that never happened in the past. And we're also starting to get submissions to this publication.
That's a news publication traditionally written by journalists and medical writers. But as physicians from the community who are reaching out and wanting to be a part of it as well. So again, it feels really good and the momentum is growing every day. So your turn leidy's going to tell you what you guys all have to do now. Actually, I wonder, can I throw a wrench in the system.
Oh, I love a wrench OK I know I knew you would. So let's take Jennifer's example and look back at this diagram. So the impact that you wanted to achieve was getting emergency medicine professionals to consult your resources over Reddit. Yes wouldn't you say that. Yes correct. And the outcome that you're looking for is increasing usage of your product, your service, your website.
So the outputs that drove that were like, what were the deliverables. Like an updated site or an app or what was the thing you changed. So we refreshed our site. We updated our podcasts, our webinars. We also grew our social media presence. We also extend it into other social media areas. So blue sky, we look at our engagement on those things.
We also started a LinkedIn again. We started to extend where our footprint was and we were able to see from there that the vibe was growing. So that's a ton actually of outputs and a ton of activities and a ton of individual inputs to make all those things happen. So let me just pause and make sure. Are we clear the difference between an outcome and an output. An output is a deliverable.
An outcome is the change in your customer's behavior and full confession. I still am not sure myself. And this is a work in progress. And that's why I think it's this conversation is so important and valuable though again, because then you can more eloquently make your case to your community, to your board, whoever it is that you're working with. And I can call you whenever I want and ask if I'm on the right track.
And let me plug also the book that we used as inspiration that I think Doctor Schiffman mentioned very briefly yesterday, which is Joshua Simon's outcomes over output. I believe it's from 2019 originally, but it was re released in 23. So reading for your trip home potentially. So now moving on to our group activity. What you have at your tables is one scenario.
Now we have two scenarios in circulation. So every table has one of those two scenarios. And you have some paper, some post-its, some markers, some things. You probably also have your own pens and paper. Feel free to use whatever you wish. Our amazing shepherd back there can help if you need more supplies. But that's what the little half sheets are is your scenario.
Now your group is going to read the scenario. Maybe one of you reads it aloud and I know it might get a little loud in here. So let's just do our best. Read the scenario and talk about what you think might be one outcome that would help achieve the goals outlined in the scenario. And I think, have already provided one output. But talk about what other outputs might help you achieve that outcome.
And then at the end of your discussion, we're going to take about 10 minutes for this, the end of your discussion. Pick someone to report back and tell us what was your one outcome statement and your one output statement. So you are either focused on the scenario about enhancing an existing peer review workflow tool, or you're looking to improve the user login experience of a website.
Single sign on implementation. Is this sound good to everyone. Are we up for a little adventure with our groups. Any questions. There's no wrong answer. As Jennifer said earlier, this is hard. This is all I am is a shepherd. So I'm helping them facilitate their meetings. And I get confused all the time.
So thank you. No wrong answers. We're still learning this stuff. So we are all in learning mode. So there's no wrong or right. Absolutely OK, go 10 minutes on the clock. Enjoy folks. Whoops okey dokey. All right, it's yours.
Do you want me to move the microphone from table to table. Yeah, that's a good idea. OK, why don't we do so. I think this is part of the session where you get to present. OK OK. Do you want to read out. Yeah OK. We're going to start with scenario one.
So scenario one you are enhancing an existing peer review workflow tool. Example output release three new features to the peer review system by the end of the quarter, including inline commenting, reviewer tagging and bulk assignment tools. Your challenge together with your group, write your goal in terms of an outcome or outcomes instead of outputs.
Take some time to debate and refine it or even to draft multiple ones. Remember, outcomes should measure the value that is delivered to either the customer or your organization. So that being said, the groups do we want to do the first one group. Yep oh, we love a volunteer. That's right.
Thank you for volunteering. I like to get it over with. So we came up with two outcomes. More efficient, i.e. shorter peer review process and reduced turnaround time of peer review process. Great and that's it. Thank you. Oops somebody else with this one.
Oh, OK. We know there's at least four tables with this. Let me out of it. Hi So one thing we wanted to mention upfront was qualitative data can sometimes be output or outcomes. So that was an interesting conversation we had. But as far as what we thought was important for outcomes, we thought about who are the people that are receiving the value from this.
We talked about editors, reviewers and authors ideally would all benefit and ways we would know if their behaviors had changed. Include more submissions from authors because the reviewers and editors are having a better experience. And so they're feeling that too, and they're submitting more. User satisfaction amongst those groups maybe easier to recruit reviewers and editors.
And yeah, I think that's what we talked about. Can you say more about the qualitative data being. Yeah tell me more about qualitative data being either outcome or outcome. Yeah well this table also mentioned that their outputs were faster turnaround times. And when we talked about something like that we said but that could also be an output or an outcome depending on what if people find that valuable like the speed and improving the speed could come at the cost of quality sometimes.
So when we think about outcomes as or outputs as quantitative data, more submissions faster. You have to make sure that you consider the quality and the experience that people have with those numbers, with that improvement, quote unquote, improvement in numbers. So I think you're on to something. One other thing we discussed was you have to be able to measure outcomes.
So we talked about how can we measure higher satisfaction with the authors, editors, reviewers. You can do that in a number of ways. You can have user groups, you can have surveys and you can measure that. And obviously more submissions is a very measurable thing. You see that right off the bat. What you've done is working well. So I would totally agree.
I would say, hey lady, can you move closer to the mic. Sorry I think you both tables are on the right track on the outcomes. Faster less friction. Increased submissions. Those are behavioral things. These are things that people are doing or not doing. The output is the deploying the survey, for instance, or skipping a step in the process.
Or if someone's logged in before saving their data from the last time, the output is more tangible. It's that deliverable piece that helps you achieve the outcome. But the outcome is changing human behavior. The outcome is much more squishy, but it should be, as you say, measurable. It should be something we can look at qualitative or quantitative data or a combination and say we are moving the needle in how people are behaving.
So that's great. You guys are doing awesome. Next scenario one table. Oh, one more on this one. Oh, do you want us to help Susan. Yeah she's like, I've got two mics, but only two feet. You can. Yeah all right. This will sound a little familiar.
So we came up with the outcomes for better quality reviews produced in less time. And we did discuss also, there's various customers of a peer review process. Authors, reviewers. Publishers, the readers. So we're looking for less friction between authors and all of the parties involved. The editors, reviewers and publishers improve perception of the journal by authors and reviewers and everyone involved.
More seamless transitions for everyone. All of the handoffs and authors and reviewers would be more excited to participate with the journal as either AI want to publish with this journal because I see they're going to go to print really quickly. I want to review for this journal because it's seamless and they really don't hassle me and it's easy to use. So those would be the output, the outcomes for the peer review.
So could I ask you to pick one of those and then tell me one output that would drive that outcome. One of the outputs would be an AI tool for taking away a lot of the grunt work on various steps. So in the article submission steps and as the review steps, the assignment steps, probably a lot of opportunity for AI. And I know everyone is excited and all that about it, but there are some really practical applications because as everyone probably knows, there's a lot of grunt work involved.
Routine, seem mundane and that's a perfect opportunity for a systematic solution for it. OK, so we're using AI to decrease friction. How are we measuring that. There is less friction or that time to publication. Yeah would be one key. I mean, there's lots of measures. But getting to publication quicker benefits everybody. I think everyone would like to but improving the quality at the same time, which is tricky to the Boeing example, sometimes it's hard to do both.
But in this scenario that we're talking about we could do both. That's excellent. Any other table do scenario one. I know we have one empty table so no. Are we ready to move on to Emily. Oh, gotcha. So, Emily, you want to talk. To John John all right. I know you're not shy.
We talked a lot about time with the outcome as well, as well as many of the other statements that the other tables have said. But it's not just for it's for really every stakeholder involved. It's the author, it's the editor, the managing editor and the staff and the reviewers. The bulk assignment, and time is measured both the individuals time doing a task and also the overall time a process takes.
And I think that's where the real value comes. So the editor with the bulk assignment can just one editor leaves, they can reassign 100 manuscripts at once instead of having to go through one by one. So it's really the time, the time savings you have, which is so easy to measure, time conversion, reducing time, increasing time, what have you. Yeah I guess I'm thinking about your comment about vibes, and I'm trying to draft an outcome in terms of a vibe.
So it's almost like a pie in the sky, what is your ideal outcome. And I think in this case, you could say, and I'm kind of going a little rogue for my group because we didn't draft finally. OK, I'm going to hand it over to my teammate in a moment. But I think one thing, one way you can draft the outcome is you want to make your journal the premier in your field, or if it's one journal among a huge portfolio, you want to make it easier and you can just say easier.
And along the way, you've got outcomes that relate to ease of author submission, ease of reviewer participation, ease of whatever, and all of those things are measurable under the general umbrella of ease. So I would say that two outcomes are, being the best journal in your field or having it be an easy product in your portfolio. So those sound like impacts to me.
The best journal to be premier, to increase revenue, to increase vibe, whatever. The kind of big picture is. And then from there we develop the outcomes that drive that. So the best journal. Well, maybe there's 10 outcomes that help define what the best journal looks like. Then there's multiple outputs that drive those outcomes. It's like a sort of hierarchical feeding system.
Yeah I was teasing Susan that in my graduate work right now, the difference between impact and outcome is pretty important because you can have short, medium, and long term outputs and short, medium, and long term impact. And how one measures impact. It's almost impossible. But I did write a statement. Our goal is to reduce the cycle time for authors, in particular those whose first language is not English, to identify specific items for revision in a manuscript rather than hunting through the manuscript for examples.
And I think Mr. Fagin's point about it being measurable is really important, because it's when you get into impact measurement, there really is not a lot of measurement going on. But I did make a note that I think there should be a vibe score, that maybe we could put a vibe score in a Likert scale and a vibe score of a five. That's pretty measurable impact factor.
It's the vibe score. Yeah, that could be on the committee, the task force, to help put that together, because I really do believe in that vibe score. And also as we're talking about all of this, I've been involved in many projects like this over my career, and it's important to do it right for the processes of it and to improve things.
But then also then you're involving people along the way. You are building that community. You're making them part of the solution and integrating them. But then it becomes challenging for me to explain to any boss I've ever had while reviving out. And they're like, yeah, that's really great. But yeah, yeah. And I'm like, look, I mean, you can feel this. They're like, no, we can't feel it at all.
So yeah, but the community can and I know it. So it's that challenge of bottling that up and measuring that. Can I ask one more question before I'm sorry to hog, but I was just curious. I came in a little bit late. Did you talk about the gestalt here. Why it's important for an organization to make a distinction between, say, an impact and an output.
Is it all is it all coming down to this notion of measurability for stakeholders or what's the purpose. Great question. I think the Boeing example helps why it's important because if we just focus on deliverables, we could be delivering features and products, that aren't achieving the goal that aren't focused on either the outcomes or the impact that we're looking to achieve.
So yeah, if that wasn't clear, thank you for underscoring that, I do. I think that was part of what Doctor Schiffman was saying yesterday in the keynote as well is we have to not worry about how many articles, or how many journals or how many hits or how many citations. Those things are useful metrics. They could be leading or lagging indicators of achieving our outcomes, but we can't measure impact with one statistic.
So yeah, that was great. Awesome awesome work. Scenario two. Yes sorry. OK. OK scenario two you are improving the user login experience for a journal website. I am going to just give one example output because that helps me to here as well implement a new single sign on, so option and password reset functionality by the end of the quarter.
So do we have any volunteers. Oh we love a run Susan. Run I'm kidding. I'll take a mic back to the other two. Thank you. My name is Katie Duffy. I'm from the seismological society. We did start a little bit with of brainstorming on outputs and then developed what we thought the outcomes would be of those outputs that we thought were good ideas.
And I will read the outcome statement first and then tell you of how we got there. So readers, users will benefit from a more inclusive journal content reading experience, with more people coming to the site directly and happier members, readers, and staff. The journal could also benefit from higher dissemination and outreach. And that's our hopeful outcome. The outputs that we thought could feed into that are number one, an accessibility audit and really taking those recommended actions and putting them into the website, as well as updated contact information, support information for anybody coming to the site.
We thought that an important measurable would be that customer service calls actually decrease and the experience the happiness increases. Happy member satisfaction survey numbers increase that we would see higher downloads and fewer turn aways if people were having an easier time with their login experience. And hopefully, this would lead to I said, higher dissemination, outreach and citations.
Really awesome. Yeah, that was great. That was great. I think I got a mic over in the table in the back. So yes, Susan you're next. Yeah great answer. We had a lot of similar thoughts as far as accessibility. So our output was actually making it more accessible.
And so the outcome for that would have been again like higher user satisfaction, but then also just reaching a broader audience because you're making your site more accessible. So you're going to have more people that are able to come look at your content. The other thing that we talked about quite a bit at the table was value. So just enhancing the value for both the publisher and the user.
So we were saying if you have a single sign on, maybe you get access to other content or you get access to email alerts or things like that. But then also from the publisher side, you're capturing this customer data so you could come up with segmentations, hyper targeting really digging into who your audience is and what content that they're looking at. So that's the two ways that we looked at this scenario.
Anything else either of you want to add. OK yeah I think that's it. Thank you. That's great. And I love the use of the other v word. I think that's a hard one to wrap your arms around, but it's super important. And that's value. So the vibe and the value, I think those go hand in hand, but they're challenging to.
But the way you described it was so eloquent. So I can't wait to go back and listen to this recording because I think that was spot on. So if I was going to diagram the thoughts that table came up with, the impact, for example, would be a broader audience. The outcomes that help us achieve that impact is having a fully compliant login experience, based on WCAG standards and then the output, the outputs that drive, that would be things like the accessibility audit, updates, to content architecture, what have you.
So that was awesome. That was a great dissection of those things. OK I think this table was next. Yeah so we use the same example output and focused on single sign on. The outcomes that we looked at were better user experience or user satisfaction, and maybe measured by reduction in cognitive load for our users.
So not having to think at all about logging in would be an ideal outcome for them if they didn't have to remember a password or think, have I visited this site before. Do I have a login for this thing. Increased engagement for us to monitor on the site. Membership considerations. Tying a membership to the journals or the products that you have access for us, getting better user data about our users and as others mentioned, maybe a reduced call volume or a reduction on customer service.
And I guess I'll just add to the more abstract conversation that as I'm a product manager personally. So I think about the differences between outputs and outcomes quite a bit. And I think this is maybe overly reductive, but what helps me is the outputs might be what that you're building the feature and the outcomes might be the why. To know with limited resources that we all have, we could be building lots of things.
What's going to make us know that we're successful in building this feature. That's the outcome for me. So yeah. Thanks that was very helpful. Can we steal that. Yeah that was brilliant. That was brilliant. OK Other tables haven't gone yet I'm sorry.
I'm losing track. Yeah, that's what I thought. I thought were a scenario one. OK I'm sorry. OK have all the know where sorry I was. We were having a little reunion. OK we're good. Everybody's done. They're done.
They're reporting back. OK, we're going to try a speed round. This is a quiz round. We have two exams 2 new scenarios for you. And we're going to as a group pick out outputs and outcomes and impact statements potentially as well. So this example is from every library. You all may have had some increased email traffic from them lately given all the threats to democracy and literacy.
So this sorry, I'm thinking about how to say that without sounding awful. You're in a safe space. Yeah OK. Thank you. So this is a slide in one of their annual reports or last year's annual report. And the headline, I would say, strengthening the connections between libraries and readers is an impact.
It's really hard to measure strengthening the connections between libraries and readers. How in the heck do I measure that. What every library has done is break down activities around banned books. Banned books week. The panels that were hosted. The authors and the publishing professionals that were brought together, the experts, the conversations that they facilitated.
So I know this is a lot of text on the screen, so apologies. But what might be an outcome that every library is looking to help achieve the strengthening connections between libraries and readers. Just shout it out. Well, no actually, because it's getting recorded. Sorry, sorry. Driving engagement between readers and the library.
So bringing people into the library for these events. And you would measure that based on circulation or circulation or I mean, you could have one of these events and then measure like if books by that particular author were checked out, or visits to the library website about the event. Nice and that's measuring a human behavior.
Yes so spot on. Absolutely spot on. Yeah Susan. I'm sorry. Over here. So I was thinking maybe increasing education and understanding around the subject of banned books and that kind of stuff where you could measure it with, like a survey, you could do, like, open forums, you could do.
I feel a ton of different measurables, but just increasing that education and understanding censorship amongst the community and the stakeholders, and the voters within the community. So that would be an outcome. Yes sorry. Did you say output. Yeah oh my God. Sorry and you would and you would measure that through.
Yeah like you could do like community surveys. You could do feedback. You could send it to the attendees of the different sessions. And then with the sessions themselves, be the outputs that are helping to deliver that. And of course, all the surveying, communications, and marketing and all that good stuff. OK yeah.
Anybody else. OK here's another one. This is from the Komen foundation. I thought this one was really interesting because it's focused on the research investments by the Komen foundation in 2023. And their statement is that research is key to developing better, more effective treatments and to understanding why some cancers stop responding to treatment and spread.
So is that statement helping to encapsulate the impact goal or is it an outcome. What does everybody think. It's hard. We can see they're measuring using the research dollars invested and the percentage of what the research focusing on the most deadly and aggressive breast cancers, and the percentage of research that addresses solutions to inequities in breast cancer.
So we know what they're measuring. But what's the outcome here. And that 40% to me too. To me, that's where I have a big question, because that's super important. And looking at that number and what it represents is very important. But to me that does not connect back to the statement here. So I would love if anyone has insight there.
And I know we have someone in the back ready to go. Go for it. Definitely agree with you Jennifer. I think for me both of these are outputs. Obviously the research is fundamental to then actually addressing the inequities. Seeing the rates of breast cancer go down in certain populations or seeing the overall rates go down or seeing earlier diagnosis or again, some other impact that would actually be measurable in the patient population.
Yeah, I would agree as well. I think that 40% on the 65, both of those are metrics of the amount of research they funded that did those things. But is that where it stops. No right. That's the output. The outcome is actually moving the needle on developing better, more effective treatments.
And to understand why cancers stop responding to treatments and care. And the connection between those two things is really important, because if all we care about is funding research, besides the fact that half of it's being taken away, but regardless. If we're only caring about delivering research, then we're missing the impact that I think all of us came to scholarly publishing to help achieve right the impacts of improving people's lives, right.
Those are hard to measure. So we need these incremental steps of what are we doing, what are we delivering and why. And then can we continue to measure that. This research is actually moving the needle. Yeah we got one over here. And then this table here in the middle. Oh, hi. So what occurred to me when you were speaking was these are outputs as stated, but without the outcomes.
How do you lobby for or get more money to do it next year or continue with your work or you have to have the outcomes to validate that in a way or to make accessible and explain why what you found is important and tie it back to that original statement. So that's.
Yeah, that's the part that I'm struggling on to. Come back. Sorry so it appears that these are hoped for outcomes. I mean, we don't really know if they'll come true or not. It's certainly the goal. But, these outputs are certainly necessary to get there.
But until we actually see the results of those outputs, these are hoped for outcomes. Yeah Well and that goes back to the leading and lagging indicators right. And I know that's business speak. But a leading indicator is a metric of what might happen. Meaning we have one of the examples in the Sidon book is a retail clothing company wants to increase return customers.
They want you to come back and buy again. They have found through their leading indicators that the more social shares, more social media. Hey, I bought this thing from North Face or whatever. That increased users coming back, customers coming back. So their outcome or their impact goal is to increase their return customers. And they're going to be able to measure that by social shares, measure that by sales, and be able to connect those to the complicating part is using the data we have now to project or predict what might happen.
And then we have to continue to gather the data to prove our point. If anyone's worked in software development. Product managers in the room. Agile development is really focused on, quick experiments, quick iterative responses, delivering value in small places, in small ways that add up to a better experience. And I think we can do that across all divisions.
It doesn't have to be software development. We have an experiment that we want to improve, the vibe. We want to bring emergency, professionals. To Jennifer's site rather than to Reddit. So in order to do that, we have to run some experiments. We have to try some things and what's working and then respond to that in a pretty agile way. So how do then we convince our organizations to let us run with those experiments.
What would be your best advice on doing that. I think breaking down the goals, breaking down the impact. You want stakeholders to increase revenue. You want to increase traffic. You want renewals. So in order to do that, I'm going to run an experiment on an outcome, an output that I think will drive an outcome that will help us deliver that impact.
So how you convince stakeholders to let you run that experiment is a great question, I don't know ideas from the room. How would you convince your boss to let you run an experiment and test an outcome to help them achieve their goal. And just so in my case, I would just do it. But I don't necessarily advise that. Yeah where who's talking.
Yeah and there's a hand. Yeah I think sharing case studies would be if there are case studies available, that's usually pretty persuasive because leaders tend to say, what are people in our vertical doing. And so on. So if you can point them to something specific, it's been efficacious. That's always partially worked for me.
Yeah Absolutely and even show them those lagging indicators. And well we want to move the needle here. We're going to run an experiment. And you tell me in three months if we've moved the needle OK. One last comment. And then we're going to wrap there in the back. Terry, you couldn't do that when I was standing next to you. You're getting your steps in, Susan. So yeah, I'm a project manager by trade.
And for me, I always ask the why. Sometimes it's not evident. And the stakeholders or whoever is going to pay for this need the problem statement or there is no project or there's no project funding, there is no experimentation. You have to have a why. And that's the first question you ask, honestly. Yeah, absolutely.
And that why needs to not be so big that we can't measure it, which is why. Why talking about impact versus outcome versus output is so important. Working working off of what Tara just said, the question I ask people who come to me and say have to do this marketing or do this on the website, is I don't need the solution. Yeah, tell me what you're trying what you're trying to ultimately achieve, and then we'll get to what the best solution is.
So that goes very much with what she said. And actually this model does unlock creativity in our organizations. We need to go there. I don't care how you get there, but we're going there is much more inspirational than I need you to deliver 14 widgets. It's like, well, OK, widget one, widget 2. All right.
Thank you all for being so brave and working together.
Segment:0 .
Hi, everyone. Thank you so much for being here today. We're so delighted to have you in the room with us, and thanks also for loving us through it. As we got you seated at these tables, we'd like to welcome you to I measure with my heart outcomes over outputs also during this session. Not on the slide and we probably should have put her on it as Susan Wilner.
And she is the shepherd of the session. And if you need any help with an extra pen or paper or anything like that. And she's been wonderful. Thank you Susan. Also, one of our co-panelists is Gabe herb, an amazing friend of Lete and mine, and also one of the most brilliant people we know. Unfortunately, he was unable to make it here today, but he was a full on participant in the shenanigans to put this session together.
And I have to say, we're going to miss our planning calls for this. My name is Jennifer rogala. I'm an associate director of publishing at Wolters Kluwer health, and this, I will just start by saying that this conversation was born from me telling Gabe and then Lete that I like to measure with my heart. Unfortunately, though, my employer doesn't agree and does like to have some measurable items coming out of my work, so I'll just leave it there and let Lete introduce herself.
Cool thank you. And I'm Lete Conrad, product experience architect at leblancs. And you are absolutely in for a treat. As you heard in the keynote yesterday, Doctor Schiffman mentioned the need to switch our minds from thinking about outputs, deliverables to outcomes. The bigger picture, and that's where you'll find your heart is really looking at that bigger picture.
So with that, let me launch right into a couple of introductory comments. So our code of conduct, I'm sure you've heard in other sessions really important to remember. We're committed to diversity, equity, and providing a safe, inclusive, and productive meeting environment. So avoid harassment. Avoid discrimination.
Avoid hostility, avoid anti-competitive. Comments just be kind. Be kind. Let's do that. And these are the values that SSP. As you've heard over and over, I'm sure community adaptability integrity and inclusivity. So what we're going to do today is think about some abstract concepts and think about some hypotheticals to apply to those concepts.
But every single one of us in this room has some need to think about how we measure what we do. So I'm hoping that you've brought your thinking caps today. I'm hoping you're up for a thought experiment. The table that you're sitting at is going to be a group that we will put together in just a few minutes, but let me start with a couple of basics.
So what do we mean when we say output. Basically dictionary definition for merriam-webster is something produced, an act, or a process or instance of producing. It's a deliverable. It's the thing I think most of us are really focused on every day in our jobs. I've got to tick that box. I've got to complete that task.
I've got to deliver the thing. The outcome is what is affected by our outputs. The outcome is actually a change in human behavior that drives business results. Or another way of putting it is it's a change in how our users, our customers, our stakeholders, our colleagues experience what we're doing in that it delivers what we're aiming for.
So we're looking to instead deliver the thing right, tick all the boxes, complete all our tasks. Actually, we need to go a level up and think about changing the behavior and the experience that we're delivering, the behavior of our customers and our users, and the experience that we're facilitating. Now, this is from the Kellogg model, which if any of you have worked in nonprofits, you may have encountered this. This was developed as a way to help organizations think about funneling all the various resources, fundraising materials, donations, et cetera into the things that will help them achieve their goals.
So nonprofits are often very focused on big picture things. Reducing a spread of a disease. Increasing access to clean water. But getting to that impact requires a whole step of processes, the inputs alone. Our donations from fundraising that isn't going to get better access to clean water, that isn't going to necessarily reduce the spread of disease.
The activities that are made possible by those inputs, those activities alone are not going to deliver on what we hope to achieve. But the inputs and the activities together can help us deliver the outputs. The outputs are those things that we're ticking off the task or completing the things that we're delivering. And all of those together then produce the outcomes, the change that we're looking to make in our communities, the change in behavior, the change in the way we talk about things, the change in experiences, those things together add up to impact.
And I know this is really broad at the moment, but we're going to make things a little more concrete. So we're going to start to look out for examples. The rest of our time together. We're going to be talking about examples of these things. So instead of trying to define it way up here in the abstract ether, we're going to make some concrete examples from all of this.
So here this is Gabe. Imagine I'm Gabe and I am now going to show our first example. So this comes to you from Gabe. This is an example outside of our industry. This is a real world non-scholarly example, Boeing's 737 MAX aircraft. The context is in the late 1910s, Boeing aimed to compete with Airbus fuel efficient A320neo by developing the 373 max, the 737 Max.
The company's primary goal was to deliver this new aircraft quickly to market. The urgency led to an emphasis on production schedules and delivery numbers. They led to a focus on outputs to meet those market demands and maintain competitiveness. Boeing succeeded in its output goal. It delivered the 737 Max within the desired timeline. So what's wrong.
This actually came at a huge cost. The focus on outputs came at the expense of critical safety considerations, which were essentially the outcomes they were hoping to deliver. The aircraft was equipped with maneuvering Characteristics Augmentation system, a new software feature that was intended to enhance flight stability. Adequate pilot training and comprehensive understanding of the software were not prioritized.
So the consequences. Two fatal crashes of the 737 Max occurred in 2018 and 2019, resulting in the loss of 346 lives. Investigations revealed that the software played a significant role in both accidents. The aftermath led to a global grounding of the 736 737 Max fleet, substantial financial losses for Boeing, and a tarnished reputation. These events the critical importance of prioritizing outcomes such as passenger safety and trust over mere outputs of production speed and delivery numbers.
So there's example number one. And then I'm going to share a scholarly publishing example. I mentioned that I work at Wolters Kluwer health. I've been there for exactly one year. I started last May. I oversee a team of publications professionals who are amazing. One of them is in the room right here, Caitlin, and we report on medical news in various specialties.
So we're going to talk about emergency medicine news today, a year ago when I started, we knew that emergency medicine news was the most read print publication in emergency medicine. We knew that from our Canter scores, we also knew, though from other surveys and from those same Canter surveys that emergency medicine physicians spent the majority of their time not with publications, not where we expected them to be.
But on Reddit. They were going to Reddit for all of their information, 40% to 50% of them. And that was their first resource. And that made me as a patient and just a human being, clutch my pearls a little bit. That was frightening. So knowing that we set out to figure out how can we build a vibe.
But again, I do work for a commercial publisher and vibes don't really drive what we do, so how could we build the relevance of this publication, make it be more than just a well read print publication, but make it be a community. Make it be. I truly believe every publication is a fingerprint. Unusual different from any other publication that's out there. And once you acknowledge and embrace that can make that publication more than that can make it into a community.
So we wanted to create that vibe, but we need outputs to deliver. Again, I can't just say, oh, it feels really good and we're vibing out over here. We had to be able to increase the relevance. We had to increase the reach of the publication. We had to increase how many people were receiving it, reading it, engaging with it. We also had to increase the outputs that we delivered as far as better, more podcasts, webinars, things like that to get the community involved.
But what have the outcomes been. One year later. The vibe is there. I can tell you our revenue is up, our relevance numbers are up. So my employer is happy. So that's good. So for anyone, some other Wolters Kluwer friends over here.
So we're doing that. But the outcomes are really good too. That vibe is happening. We've had in-person events where we've had people spilling out onto the street, and they're all based on emergency medicine news. We have multimedia events that are happening that never happened in the past. And we're also starting to get submissions to this publication.
That's a news publication traditionally written by journalists and medical writers. But as physicians from the community who are reaching out and wanting to be a part of it as well. So again, it feels really good and the momentum is growing every day. So your turn leidy's going to tell you what you guys all have to do now. Actually, I wonder, can I throw a wrench in the system.
Oh, I love a wrench OK I know I knew you would. So let's take Jennifer's example and look back at this diagram. So the impact that you wanted to achieve was getting emergency medicine professionals to consult your resources over Reddit. Yes wouldn't you say that. Yes correct. And the outcome that you're looking for is increasing usage of your product, your service, your website.
So the outputs that drove that were like, what were the deliverables. Like an updated site or an app or what was the thing you changed. So we refreshed our site. We updated our podcasts, our webinars. We also grew our social media presence. We also extend it into other social media areas. So blue sky, we look at our engagement on those things.
We also started a LinkedIn again. We started to extend where our footprint was and we were able to see from there that the vibe was growing. So that's a ton actually of outputs and a ton of activities and a ton of individual inputs to make all those things happen. So let me just pause and make sure. Are we clear the difference between an outcome and an output. An output is a deliverable.
An outcome is the change in your customer's behavior and full confession. I still am not sure myself. And this is a work in progress. And that's why I think it's this conversation is so important and valuable though again, because then you can more eloquently make your case to your community, to your board, whoever it is that you're working with. And I can call you whenever I want and ask if I'm on the right track.
And let me plug also the book that we used as inspiration that I think Doctor Schiffman mentioned very briefly yesterday, which is Joshua Simon's outcomes over output. I believe it's from 2019 originally, but it was re released in 23. So reading for your trip home potentially. So now moving on to our group activity. What you have at your tables is one scenario.
Now we have two scenarios in circulation. So every table has one of those two scenarios. And you have some paper, some post-its, some markers, some things. You probably also have your own pens and paper. Feel free to use whatever you wish. Our amazing shepherd back there can help if you need more supplies. But that's what the little half sheets are is your scenario.
Now your group is going to read the scenario. Maybe one of you reads it aloud and I know it might get a little loud in here. So let's just do our best. Read the scenario and talk about what you think might be one outcome that would help achieve the goals outlined in the scenario. And I think, have already provided one output. But talk about what other outputs might help you achieve that outcome.
And then at the end of your discussion, we're going to take about 10 minutes for this, the end of your discussion. Pick someone to report back and tell us what was your one outcome statement and your one output statement. So you are either focused on the scenario about enhancing an existing peer review workflow tool, or you're looking to improve the user login experience of a website.
Single sign on implementation. Is this sound good to everyone. Are we up for a little adventure with our groups. Any questions. There's no wrong answer. As Jennifer said earlier, this is hard. This is all I am is a shepherd. So I'm helping them facilitate their meetings. And I get confused all the time.
So thank you. No wrong answers. We're still learning this stuff. So we are all in learning mode. So there's no wrong or right. Absolutely OK, go 10 minutes on the clock. Enjoy folks. Whoops okey dokey. All right, it's yours.
Do you want me to move the microphone from table to table. Yeah, that's a good idea. OK, why don't we do so. I think this is part of the session where you get to present. OK OK. Do you want to read out. Yeah OK. We're going to start with scenario one.
So scenario one you are enhancing an existing peer review workflow tool. Example output release three new features to the peer review system by the end of the quarter, including inline commenting, reviewer tagging and bulk assignment tools. Your challenge together with your group, write your goal in terms of an outcome or outcomes instead of outputs.
Take some time to debate and refine it or even to draft multiple ones. Remember, outcomes should measure the value that is delivered to either the customer or your organization. So that being said, the groups do we want to do the first one group. Yep oh, we love a volunteer. That's right.
Thank you for volunteering. I like to get it over with. So we came up with two outcomes. More efficient, i.e. shorter peer review process and reduced turnaround time of peer review process. Great and that's it. Thank you. Oops somebody else with this one.
Oh, OK. We know there's at least four tables with this. Let me out of it. Hi So one thing we wanted to mention upfront was qualitative data can sometimes be output or outcomes. So that was an interesting conversation we had. But as far as what we thought was important for outcomes, we thought about who are the people that are receiving the value from this.
We talked about editors, reviewers and authors ideally would all benefit and ways we would know if their behaviors had changed. Include more submissions from authors because the reviewers and editors are having a better experience. And so they're feeling that too, and they're submitting more. User satisfaction amongst those groups maybe easier to recruit reviewers and editors.
And yeah, I think that's what we talked about. Can you say more about the qualitative data being. Yeah tell me more about qualitative data being either outcome or outcome. Yeah well this table also mentioned that their outputs were faster turnaround times. And when we talked about something like that we said but that could also be an output or an outcome depending on what if people find that valuable like the speed and improving the speed could come at the cost of quality sometimes.
So when we think about outcomes as or outputs as quantitative data, more submissions faster. You have to make sure that you consider the quality and the experience that people have with those numbers, with that improvement, quote unquote, improvement in numbers. So I think you're on to something. One other thing we discussed was you have to be able to measure outcomes.
So we talked about how can we measure higher satisfaction with the authors, editors, reviewers. You can do that in a number of ways. You can have user groups, you can have surveys and you can measure that. And obviously more submissions is a very measurable thing. You see that right off the bat. What you've done is working well. So I would totally agree.
I would say, hey lady, can you move closer to the mic. Sorry I think you both tables are on the right track on the outcomes. Faster less friction. Increased submissions. Those are behavioral things. These are things that people are doing or not doing. The output is the deploying the survey, for instance, or skipping a step in the process.
Or if someone's logged in before saving their data from the last time, the output is more tangible. It's that deliverable piece that helps you achieve the outcome. But the outcome is changing human behavior. The outcome is much more squishy, but it should be, as you say, measurable. It should be something we can look at qualitative or quantitative data or a combination and say we are moving the needle in how people are behaving.
So that's great. You guys are doing awesome. Next scenario one table. Oh, one more on this one. Oh, do you want us to help Susan. Yeah she's like, I've got two mics, but only two feet. You can. Yeah all right. This will sound a little familiar.
So we came up with the outcomes for better quality reviews produced in less time. And we did discuss also, there's various customers of a peer review process. Authors, reviewers. Publishers, the readers. So we're looking for less friction between authors and all of the parties involved. The editors, reviewers and publishers improve perception of the journal by authors and reviewers and everyone involved.
More seamless transitions for everyone. All of the handoffs and authors and reviewers would be more excited to participate with the journal as either AI want to publish with this journal because I see they're going to go to print really quickly. I want to review for this journal because it's seamless and they really don't hassle me and it's easy to use. So those would be the output, the outcomes for the peer review.
So could I ask you to pick one of those and then tell me one output that would drive that outcome. One of the outputs would be an AI tool for taking away a lot of the grunt work on various steps. So in the article submission steps and as the review steps, the assignment steps, probably a lot of opportunity for AI. And I know everyone is excited and all that about it, but there are some really practical applications because as everyone probably knows, there's a lot of grunt work involved.
Routine, seem mundane and that's a perfect opportunity for a systematic solution for it. OK, so we're using AI to decrease friction. How are we measuring that. There is less friction or that time to publication. Yeah would be one key. I mean, there's lots of measures. But getting to publication quicker benefits everybody. I think everyone would like to but improving the quality at the same time, which is tricky to the Boeing example, sometimes it's hard to do both.
But in this scenario that we're talking about we could do both. That's excellent. Any other table do scenario one. I know we have one empty table so no. Are we ready to move on to Emily. Oh, gotcha. So, Emily, you want to talk. To John John all right. I know you're not shy.
We talked a lot about time with the outcome as well, as well as many of the other statements that the other tables have said. But it's not just for it's for really every stakeholder involved. It's the author, it's the editor, the managing editor and the staff and the reviewers. The bulk assignment, and time is measured both the individuals time doing a task and also the overall time a process takes.
And I think that's where the real value comes. So the editor with the bulk assignment can just one editor leaves, they can reassign 100 manuscripts at once instead of having to go through one by one. So it's really the time, the time savings you have, which is so easy to measure, time conversion, reducing time, increasing time, what have you. Yeah I guess I'm thinking about your comment about vibes, and I'm trying to draft an outcome in terms of a vibe.
So it's almost like a pie in the sky, what is your ideal outcome. And I think in this case, you could say, and I'm kind of going a little rogue for my group because we didn't draft finally. OK, I'm going to hand it over to my teammate in a moment. But I think one thing, one way you can draft the outcome is you want to make your journal the premier in your field, or if it's one journal among a huge portfolio, you want to make it easier and you can just say easier.
And along the way, you've got outcomes that relate to ease of author submission, ease of reviewer participation, ease of whatever, and all of those things are measurable under the general umbrella of ease. So I would say that two outcomes are, being the best journal in your field or having it be an easy product in your portfolio. So those sound like impacts to me.
The best journal to be premier, to increase revenue, to increase vibe, whatever. The kind of big picture is. And then from there we develop the outcomes that drive that. So the best journal. Well, maybe there's 10 outcomes that help define what the best journal looks like. Then there's multiple outputs that drive those outcomes. It's like a sort of hierarchical feeding system.
Yeah I was teasing Susan that in my graduate work right now, the difference between impact and outcome is pretty important because you can have short, medium, and long term outputs and short, medium, and long term impact. And how one measures impact. It's almost impossible. But I did write a statement. Our goal is to reduce the cycle time for authors, in particular those whose first language is not English, to identify specific items for revision in a manuscript rather than hunting through the manuscript for examples.
And I think Mr. Fagin's point about it being measurable is really important, because it's when you get into impact measurement, there really is not a lot of measurement going on. But I did make a note that I think there should be a vibe score, that maybe we could put a vibe score in a Likert scale and a vibe score of a five. That's pretty measurable impact factor.
It's the vibe score. Yeah, that could be on the committee, the task force, to help put that together, because I really do believe in that vibe score. And also as we're talking about all of this, I've been involved in many projects like this over my career, and it's important to do it right for the processes of it and to improve things.
But then also then you're involving people along the way. You are building that community. You're making them part of the solution and integrating them. But then it becomes challenging for me to explain to any boss I've ever had while reviving out. And they're like, yeah, that's really great. But yeah, yeah. And I'm like, look, I mean, you can feel this. They're like, no, we can't feel it at all.
So yeah, but the community can and I know it. So it's that challenge of bottling that up and measuring that. Can I ask one more question before I'm sorry to hog, but I was just curious. I came in a little bit late. Did you talk about the gestalt here. Why it's important for an organization to make a distinction between, say, an impact and an output.
Is it all is it all coming down to this notion of measurability for stakeholders or what's the purpose. Great question. I think the Boeing example helps why it's important because if we just focus on deliverables, we could be delivering features and products, that aren't achieving the goal that aren't focused on either the outcomes or the impact that we're looking to achieve.
So yeah, if that wasn't clear, thank you for underscoring that, I do. I think that was part of what Doctor Schiffman was saying yesterday in the keynote as well is we have to not worry about how many articles, or how many journals or how many hits or how many citations. Those things are useful metrics. They could be leading or lagging indicators of achieving our outcomes, but we can't measure impact with one statistic.
So yeah, that was great. Awesome awesome work. Scenario two. Yes sorry. OK. OK scenario two you are improving the user login experience for a journal website. I am going to just give one example output because that helps me to here as well implement a new single sign on, so option and password reset functionality by the end of the quarter.
So do we have any volunteers. Oh we love a run Susan. Run I'm kidding. I'll take a mic back to the other two. Thank you. My name is Katie Duffy. I'm from the seismological society. We did start a little bit with of brainstorming on outputs and then developed what we thought the outcomes would be of those outputs that we thought were good ideas.
And I will read the outcome statement first and then tell you of how we got there. So readers, users will benefit from a more inclusive journal content reading experience, with more people coming to the site directly and happier members, readers, and staff. The journal could also benefit from higher dissemination and outreach. And that's our hopeful outcome. The outputs that we thought could feed into that are number one, an accessibility audit and really taking those recommended actions and putting them into the website, as well as updated contact information, support information for anybody coming to the site.
We thought that an important measurable would be that customer service calls actually decrease and the experience the happiness increases. Happy member satisfaction survey numbers increase that we would see higher downloads and fewer turn aways if people were having an easier time with their login experience. And hopefully, this would lead to I said, higher dissemination, outreach and citations.
Really awesome. Yeah, that was great. That was great. I think I got a mic over in the table in the back. So yes, Susan you're next. Yeah great answer. We had a lot of similar thoughts as far as accessibility. So our output was actually making it more accessible.
And so the outcome for that would have been again like higher user satisfaction, but then also just reaching a broader audience because you're making your site more accessible. So you're going to have more people that are able to come look at your content. The other thing that we talked about quite a bit at the table was value. So just enhancing the value for both the publisher and the user.
So we were saying if you have a single sign on, maybe you get access to other content or you get access to email alerts or things like that. But then also from the publisher side, you're capturing this customer data so you could come up with segmentations, hyper targeting really digging into who your audience is and what content that they're looking at. So that's the two ways that we looked at this scenario.
Anything else either of you want to add. OK yeah I think that's it. Thank you. That's great. And I love the use of the other v word. I think that's a hard one to wrap your arms around, but it's super important. And that's value. So the vibe and the value, I think those go hand in hand, but they're challenging to.
But the way you described it was so eloquent. So I can't wait to go back and listen to this recording because I think that was spot on. So if I was going to diagram the thoughts that table came up with, the impact, for example, would be a broader audience. The outcomes that help us achieve that impact is having a fully compliant login experience, based on WCAG standards and then the output, the outputs that drive, that would be things like the accessibility audit, updates, to content architecture, what have you.
So that was awesome. That was a great dissection of those things. OK I think this table was next. Yeah so we use the same example output and focused on single sign on. The outcomes that we looked at were better user experience or user satisfaction, and maybe measured by reduction in cognitive load for our users.
So not having to think at all about logging in would be an ideal outcome for them if they didn't have to remember a password or think, have I visited this site before. Do I have a login for this thing. Increased engagement for us to monitor on the site. Membership considerations. Tying a membership to the journals or the products that you have access for us, getting better user data about our users and as others mentioned, maybe a reduced call volume or a reduction on customer service.
And I guess I'll just add to the more abstract conversation that as I'm a product manager personally. So I think about the differences between outputs and outcomes quite a bit. And I think this is maybe overly reductive, but what helps me is the outputs might be what that you're building the feature and the outcomes might be the why. To know with limited resources that we all have, we could be building lots of things.
What's going to make us know that we're successful in building this feature. That's the outcome for me. So yeah. Thanks that was very helpful. Can we steal that. Yeah that was brilliant. That was brilliant. OK Other tables haven't gone yet I'm sorry.
I'm losing track. Yeah, that's what I thought. I thought were a scenario one. OK I'm sorry. OK have all the know where sorry I was. We were having a little reunion. OK we're good. Everybody's done. They're done.
They're reporting back. OK, we're going to try a speed round. This is a quiz round. We have two exams 2 new scenarios for you. And we're going to as a group pick out outputs and outcomes and impact statements potentially as well. So this example is from every library. You all may have had some increased email traffic from them lately given all the threats to democracy and literacy.
So this sorry, I'm thinking about how to say that without sounding awful. You're in a safe space. Yeah OK. Thank you. So this is a slide in one of their annual reports or last year's annual report. And the headline, I would say, strengthening the connections between libraries and readers is an impact.
It's really hard to measure strengthening the connections between libraries and readers. How in the heck do I measure that. What every library has done is break down activities around banned books. Banned books week. The panels that were hosted. The authors and the publishing professionals that were brought together, the experts, the conversations that they facilitated.
So I know this is a lot of text on the screen, so apologies. But what might be an outcome that every library is looking to help achieve the strengthening connections between libraries and readers. Just shout it out. Well, no actually, because it's getting recorded. Sorry, sorry. Driving engagement between readers and the library.
So bringing people into the library for these events. And you would measure that based on circulation or circulation or I mean, you could have one of these events and then measure like if books by that particular author were checked out, or visits to the library website about the event. Nice and that's measuring a human behavior.
Yes so spot on. Absolutely spot on. Yeah Susan. I'm sorry. Over here. So I was thinking maybe increasing education and understanding around the subject of banned books and that kind of stuff where you could measure it with, like a survey, you could do, like, open forums, you could do.
I feel a ton of different measurables, but just increasing that education and understanding censorship amongst the community and the stakeholders, and the voters within the community. So that would be an outcome. Yes sorry. Did you say output. Yeah oh my God. Sorry and you would and you would measure that through.
Yeah like you could do like community surveys. You could do feedback. You could send it to the attendees of the different sessions. And then with the sessions themselves, be the outputs that are helping to deliver that. And of course, all the surveying, communications, and marketing and all that good stuff. OK yeah.
Anybody else. OK here's another one. This is from the Komen foundation. I thought this one was really interesting because it's focused on the research investments by the Komen foundation in 2023. And their statement is that research is key to developing better, more effective treatments and to understanding why some cancers stop responding to treatment and spread.
So is that statement helping to encapsulate the impact goal or is it an outcome. What does everybody think. It's hard. We can see they're measuring using the research dollars invested and the percentage of what the research focusing on the most deadly and aggressive breast cancers, and the percentage of research that addresses solutions to inequities in breast cancer.
So we know what they're measuring. But what's the outcome here. And that 40% to me too. To me, that's where I have a big question, because that's super important. And looking at that number and what it represents is very important. But to me that does not connect back to the statement here. So I would love if anyone has insight there.
And I know we have someone in the back ready to go. Go for it. Definitely agree with you Jennifer. I think for me both of these are outputs. Obviously the research is fundamental to then actually addressing the inequities. Seeing the rates of breast cancer go down in certain populations or seeing the overall rates go down or seeing earlier diagnosis or again, some other impact that would actually be measurable in the patient population.
Yeah, I would agree as well. I think that 40% on the 65, both of those are metrics of the amount of research they funded that did those things. But is that where it stops. No right. That's the output. The outcome is actually moving the needle on developing better, more effective treatments.
And to understand why cancers stop responding to treatments and care. And the connection between those two things is really important, because if all we care about is funding research, besides the fact that half of it's being taken away, but regardless. If we're only caring about delivering research, then we're missing the impact that I think all of us came to scholarly publishing to help achieve right the impacts of improving people's lives, right.
Those are hard to measure. So we need these incremental steps of what are we doing, what are we delivering and why. And then can we continue to measure that. This research is actually moving the needle. Yeah we got one over here. And then this table here in the middle. Oh, hi. So what occurred to me when you were speaking was these are outputs as stated, but without the outcomes.
How do you lobby for or get more money to do it next year or continue with your work or you have to have the outcomes to validate that in a way or to make accessible and explain why what you found is important and tie it back to that original statement. So that's.
Yeah, that's the part that I'm struggling on to. Come back. Sorry so it appears that these are hoped for outcomes. I mean, we don't really know if they'll come true or not. It's certainly the goal. But, these outputs are certainly necessary to get there.
But until we actually see the results of those outputs, these are hoped for outcomes. Yeah Well and that goes back to the leading and lagging indicators right. And I know that's business speak. But a leading indicator is a metric of what might happen. Meaning we have one of the examples in the Sidon book is a retail clothing company wants to increase return customers.
They want you to come back and buy again. They have found through their leading indicators that the more social shares, more social media. Hey, I bought this thing from North Face or whatever. That increased users coming back, customers coming back. So their outcome or their impact goal is to increase their return customers. And they're going to be able to measure that by social shares, measure that by sales, and be able to connect those to the complicating part is using the data we have now to project or predict what might happen.
And then we have to continue to gather the data to prove our point. If anyone's worked in software development. Product managers in the room. Agile development is really focused on, quick experiments, quick iterative responses, delivering value in small places, in small ways that add up to a better experience. And I think we can do that across all divisions.
It doesn't have to be software development. We have an experiment that we want to improve, the vibe. We want to bring emergency, professionals. To Jennifer's site rather than to Reddit. So in order to do that, we have to run some experiments. We have to try some things and what's working and then respond to that in a pretty agile way. So how do then we convince our organizations to let us run with those experiments.
What would be your best advice on doing that. I think breaking down the goals, breaking down the impact. You want stakeholders to increase revenue. You want to increase traffic. You want renewals. So in order to do that, I'm going to run an experiment on an outcome, an output that I think will drive an outcome that will help us deliver that impact.
So how you convince stakeholders to let you run that experiment is a great question, I don't know ideas from the room. How would you convince your boss to let you run an experiment and test an outcome to help them achieve their goal. And just so in my case, I would just do it. But I don't necessarily advise that. Yeah where who's talking.
Yeah and there's a hand. Yeah I think sharing case studies would be if there are case studies available, that's usually pretty persuasive because leaders tend to say, what are people in our vertical doing. And so on. So if you can point them to something specific, it's been efficacious. That's always partially worked for me.
Yeah Absolutely and even show them those lagging indicators. And well we want to move the needle here. We're going to run an experiment. And you tell me in three months if we've moved the needle OK. One last comment. And then we're going to wrap there in the back. Terry, you couldn't do that when I was standing next to you. You're getting your steps in, Susan. So yeah, I'm a project manager by trade.
And for me, I always ask the why. Sometimes it's not evident. And the stakeholders or whoever is going to pay for this need the problem statement or there is no project or there's no project funding, there is no experimentation. You have to have a why. And that's the first question you ask, honestly. Yeah, absolutely.
And that why needs to not be so big that we can't measure it, which is why. Why talking about impact versus outcome versus output is so important. Working working off of what Tara just said, the question I ask people who come to me and say have to do this marketing or do this on the website, is I don't need the solution. Yeah, tell me what you're trying what you're trying to ultimately achieve, and then we'll get to what the best solution is.
So that goes very much with what she said. And actually this model does unlock creativity in our organizations. We need to go there. I don't care how you get there, but we're going there is much more inspirational than I need you to deliver 14 widgets. It's like, well, OK, widget one, widget 2. All right.
Thank you all for being so brave and working together.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode