Cadmore media player playing video The Research and Publishing Experience – View from the Front Lines
Video Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the video. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Language
1x
Play Speed
Adjust Volume
Transcript
Search
Result Count
of
Click to jump to result
Search
Re-center
Segments
Result Count
of
Click to jump to result
Search
Name:
The Research and Publishing Experience – View from the Front Lines
Description:
The Research and Publishing Experience – View from the Front Lines
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/c0aafac2-77b6-4d78-9521-b4464d14f2f3/thumbnails/c0aafac2-77b6-4d78-9521-b4464d14f2f3.jpg
Duration:
T00H55M34S
Embed URL:
https://stream.cadmore.media/player/c0aafac2-77b6-4d78-9521-b4464d14f2f3
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/c0aafac2-77b6-4d78-9521-b4464d14f2f3/The Research and Publishing Experience %e2%80%93 View from the Front Lines.mp4?sv=2019-02-02&sr=c&sig=vredOWnB6y9hShnTGNKD9SAZZZ0SiU1NCFrBHcsV7M4%3D&st=2025-01-22T13%3A37%3A48Z&se=2025-01-22T15%3A42%3A48Z&sp=r
Upload Date:
2020-11-18T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
DUSTIN SMITH: Really supposed to be like a coffee hour. Like, you basically just are awkwardly impinging upon a conversation between the core people. We're glad to have you, though. And so we're just going to talk and tell the stories. I'm going to ask some questions. But first of all, not everybody here knows each other. So they're going to introduce themselves to each other. And then we're just going to ask some questions and take it from there.
WILL FORTIN: Cool. I'm going to break the fourth wall a little bit to say hi and sorry for the people who are behind me. But yeah, I'm Will. I think I've talked to some of you on the phone and some of you not. And I'm a research scientist at Columbia University. I work at the Lamont Doherty Earth Observatory. So I'm a geophysicist doing active source marine seismology.
WILL FORTIN: So I end up writing papers, reviewing papers, reading papers, interacting with students, and postdocs, and other researchers, and you know, do the whole user side of where the sausage is made.
INGRID NELSON: Nice to meet you, Will. I'm Ingrid Nelson. I'm a doctor. I work in the Bronx at a city hospital. Pure consumer research, very occasional producer. And I'm always looking for better ways to figure things out.
DUSTIN SMITH: And what field-- what--
INGRID NELSON: So I'm an internist, and I also am a hospice and palliative care physician. So I work both outpatient and inpatient. And again, it's a city hospital I work at.
DUSTIN SMITH: What's the vibe at the city hospital, for people who don't know?
INGRID NELSON: So New York City actually has the biggest city public health system in the country. We have, I think, more than a dozen city hospitals. All comers are welcome. You know, if people can't pay their bills, that's just fine. No one gets turned away. And in the whole-- all the five boroughs, we've cobbled together a pretty good health care system. You may not get your cath in the Bronx, but you can get it at Bellevue.
INGRID NELSON: And we have transportation. So it's a very-- the population is like New York City. It's absolutely varied. And you know, it's like working in the United Nations, we say, sometimes, which is meeting now. And we're really working every day with folks who are struggling to make ends meet, are struggling to stay in the country, are struggling to stay out of jail sometimes.
INGRID NELSON: So that's the vibe.
MEGAN ANSDELL: So we have a very varied group. So I'm Megan. So I'm a research fellow at the Flatiron Institute. So I'm a computational astrophysicist. And the Flatiron Institute is sort of the in-house research arm for the Simons Foundation. And we focus on-- so the Simons Foundation is a private foundation that funds a variety of science topics, and they focus a lot on computational science. So I publish a lot.
MEGAN ANSDELL: I review papers in astrophysics. But I also interact a lot with computer scientists and machine learning people who have a very different way of publishing and sharing their work. I also have a background in science policy. So I care a lot about topics on how to engage people.
WILL FORTIN: So do you have field expeditions then? Because one of the--
DUSTIN SMITH: Into space?
WILL FORTIN: No. I mean, going to a telescope-- one of the topics in the previous session was data management and how we use that. And I think that is related to publishing, because we need to figure out what we're going to do with that and how we're going to share it. Because I go out to sea for a month or two months at a time. And it's in the North Pacific. And it's terrible, and it smells like diesel.
WILL FORTIN: But you come back with huge amounts of data. And we've got to deal with that. And we deal with it some in-house, some out. And you guys have huge data sets, I'm sure. But do you guys produce data sets if it's a largely computational facilities, or models and--
MEGAN ANSDELL: Yes.
WILL FORTIN: I mean, those are data sets. But--
MEGAN ANSDELL: So at least in our field-- I think astronomy is kind of interesting because we do-- like, we go to the telescope and we get our own personal data. You know, we've applied for time at this telescope we go. We get the data. And for a long time people kept that data on whatever format they wanted to. This data would be lost, you know, and it wouldn't be shared. But because we also have this side, where we use a lot of data from spacecraft that are funded by NASA or something like that-- and so they have to be public.
MEGAN ANSDELL: So our field has opened up a lot in that sense. So we still have ground-based observatories, where, like, an individual PI will go and gather data. But it's becoming more and more popular to have that data automatically, after, like, a year, become public in some. So we're much more organized than, I think, than other sciences. And then the Simons Foundation is very into providing big data sets that can be used for computational science and making that publicly available.
MEGAN ANSDELL: So at least where I work, it's very much encouraged to make these curated data sets and make them available to people so that everybody can use them.
DUSTIN SMITH: How do you use the data, take it all the way through publishing? Is that still a big inhibition?
WILL FORTIN: Well, yes. I mean, when you think about research-- and you know, I talk to my nieces, and nephews, and sisters, and whatever-- they think I'm-- movies have skewed it so much. They think you're having eureka moments, and you're typing on keyboards, and there's music playing. 90% of it is just managing data and, like, beating it into form where you can get it into this program or this code I wrote.
WILL FORTIN: And I work a lot in advanced computational techniques, and machine learning, and seismic inversion. So these things run on, well, supercomputers, right, or clusters of computers. And it takes 1,100 computers going for a week to come up with one result. And so we got to be really careful with our data. So we handle it so much and so carefully that it's problematic.
WILL FORTIN: Some of it's old, some of it's new. You know, you try to amass geophysical borehole data that's been collected from 1950. Nobody has the same format. So I was digging around warehouses in Cape Cod, literally dust in my nose, trying to come up with paper records of seismic data. And you can't treat that the same as modern data, you know? It's a huge problem.
WILL FORTIN: And consolidating it would be so fantastic, particularly in the face of machine learning being the future of so much of science that you need to be able to give the computers data in a format that it can deal with repeatedly, AKA no paper records, no difference between a ship from 1970 to 2012, so whatever. That's a huge problem. And it's going to take a lot of time and effort on people's part in order to get it there, in order for us to do stuff.
WILL FORTIN: And so it's really cool that you guys have standardized stuff. And I assume the hospitals are doing similar form entries and compiling records.
INGRID NELSON: Well, I'd say so, but for different reasons.
WILL FORTIN: More important.
INGRID NELSON: Yeah, well, you know, it's so we can measure outcomes and we can keep track of how much money we're losing. Because we always lose money. But medicine is funny. Because we-- well, it's not funny. Medicine-- we rely on data. And that's why you read journal articles to see what works and what doesn't work. But ultimately, it's the interaction with one person that is really at the heart of your job.
INGRID NELSON: So it's very different. It's very different. We, you know, adopt standards of care which are based on data. But our work is with one person at a time.
DUSTIN SMITH: Ingrid, can you talk more about what keeping up in your field means? And you talk about one person. Is it basically you're sort of cultivating your knowledge base at the margins, based upon your patient load for the day?
INGRID NELSON: Oh, that's, like, the dirtiest word in medicine, the N of 1, right? So we try to avoid that. We try to avoid that. So I think the big leveler in terms of-- well, in the olden days, when I went to medical school in the 1990s, everyone had Harrison's Textbook of Internal Medicine, which is about the size of an OED.
DUSTIN SMITH: [INAUDIBLE] somewhere? There we go.
INGRID NELSON: Yeah, there we go. [INTERPOSING VOICES]
INGRID NELSON: Which print is too small for me to read anymore. But it was great because-- because you looked something up. You look something up.
DUSTIN SMITH: [INAUDIBLE] the new edition right now.
INGRID NELSON: We'll talk later. So you'd look something up, and there'd be 10 citations. And two and a half hours later, you're sort of through the pathophysiology. And then you find out what probably you should be doing. And then up to date came out. And that was, like, a game-changer. And I think it had a lot of effects.
DUSTIN SMITH: Do you guys know what up to date?
INGRID NELSON: You guess know what Up to Date is? So it's a computer program, where it's a big bank of information and all different subjects in medicine written by specialists in the field with a wonderful bibliography. And so you can just go in and type out adrenal adenoma. And you used to get a very clear kind of short description and an idea of where to go next with it.
WILL FORTIN: So Wikipedia is not good enough, or WebMD?
INGRID NELSON: Well, I'm going back to Google. Going back to Google. So in standardized medicine, it made our life a lot easier. And so you could spend the five minutes you needed to figure out what to do, and then spend the rest of the two and a half minutes you're allotted with each patient actually talking to them. So that's a joke. We actually get more.
DUSTIN SMITH: [INAUDIBLE] Yeah. Go ahead.
INGRID NELSON: So yeah. So we used Up to Date for a long time. But it kind of has been a victim of its own success. I think it's gotten really much broader, and much flatter, and much harder to find things in. So funnily enough, me and many of my colleagues are kind of going back to reading articles in journals. And we find them in the bibliography of Up to Date or in Google Scholar.
DUSTIN SMITH: So instead of Up to Date, are you actually going to the Google Scholar interface and--
INGRID NELSON: Absolutely. Absolutely. Well, the bibliography of Up to Date, but then Google Scholar. Yeah.
DUSTIN SMITH: Interesting. So I mean, when do you guys think about keeping up with your field, what drives that, and how do you actually try to achieve that? I assume it's pretty futile?
WILL FORTIN: No. I probably don't read as much as I should. But I think everyone would tell you that. So I think what you should do is nebulous and false. But the way I go about it is I've got a number of papers that I think are key papers for things that my colleagues have written or I've written. And if somebody references those, it's probably relevant. And so you can skim that. So it's either a Google alert or a Web Science is typically what I use.
WILL FORTIN: So if anybody ever cites this, it pops up in my email every week of like, here's a list of papers that cited this paper, that paper, whatever. And it's pretty easy. That's the tool that I use the most-- Web Science-- to find papers.
INGRID NELSON: Who sponsors Web Science?
WILL FORTIN: Oh, geez, I should know that.
INGRID NELSON: Oh, sorry.
WILL FORTIN: Probably everybody in the room but me knows that.
DUSTIN SMITH: Hi, Chris. [INAUDIBLE]
WILL FORTIN: OK, that was my guess.
DUSTIN SMITH: The bearded gentleman over there.
WILL FORTIN: Yeah.
DUSTIN SMITH: You can--
WILL FORTIN: Had a very good interaction with your online help people the other day when I lost roaming privileges. Thank you.
INGRID NELSON: So I guess what I was asking is that really sort of unbiased and open and--
WILL FORTIN: Well, I think it seems to be. I mean, I think most people I know in our field use that. And I know you guys use something different in astro, typically. And we can talk about my reaction to that later. But it's a little slower because it takes a little while for the system, I think, to hit that this has been referenced, and it's all after peer review. And you know, most of the most pressing stuff that I need to know and what's going on here and there happens in meetings and in communities.
WILL FORTIN: And I'm lucky to be at Lamont. It's like this huge-- for the people who don't know that it's a really big Earth Science Institute at Columbia. It's 600 people. And we have so many seminars you couldn't go to all of them. And so you sit in rooms with people. And you know who's collecting what data, who's doing what, and whatever.
WILL FORTIN: And if you have a question, it's pretty easy to get a hold of somebody. So all the most cutting edge stuff is usually done that way. But I think when I was in grad school at the University of Wyoming-- it's a good geology department. A lot of rocks in Wyoming. But you don't get daily seminars there. Because who's going to go to Laramie?
WILL FORTIN: So it's nice in that way. And Web Science-- yeah, it was perfectly adequate for that. adequate for that, but [INAUDIBLE]..
MEGAN ANSDELL: To your point about saying, you know, it depends on what Institute you're at. So I did my PhD at the University of Hawaii. And although it's a beautiful place, it is like one of the most remote inhabited islands in the world. So you know we don't get a lot of people coming through.
WILL FORTIN: It's like a $600 minimum flight. Like, you're trying to get a Tuesday speaker, come on.
MEGAN ANSDELL: Yeah, not going happen. So we use the archive a lot, which is a place for preprints that focuses a lot on physics, computational science, computer science, things like that. But you know, every day you get all the papers that were published that day and you get a title and an abstract. And your sort of motivation to look at this every day is basically built on a system of like shaming each other. So people would be like, oh, did you see this paper on the archive?
MEGAN ANSDELL: And if you say no, then they're like, oh, you know. It can be academics are cut throat, especially astrophysicists.
DUSTIN SMITH: We're monsters.
MEGAN ANSDELL: Yeah. So you know, so there's journal clubs also at these institutes. So I ran a journal club when I was in grad school. And you know it happened twice a week. And you talk about whatever papers were there that day that showed up on the archive. And you can filter your archive search to only give you-- so I study protoplanetary disks. And I study planet formation. And so mine was heavily biased towards like that.
DUSTIN SMITH: Do you feel that's good enough? Is that too crude?
MEGAN ANSDELL: You know, you get-- you could-- I mean, you can obviously scan titles and if a title has catches my interest, then I'll read the abstract or something like that. But I rarely-- I'll download a bunch of papers that will sit on my desktop and I won't read them. And then I'll feel bad about deleting all of them like two weeks later when they're no longer relevant. And this cycle just perpetuates itself.
WILL FORTIN: For me it's so many tabs in Chrome.
MEGAN ANSDELL: And you never want to close your browser.
DUSTIN SMITH: When you have 64 gig of RAM. That's a big problem.
MEGAN ANSDELL: But I'm at an Institute now where people are coming through all the time, you know, and so, I'm actually getting all my news from-- I barely read the archive anymore. Because I'm just in group meetings all the time with different people. And you just hear about these things naturally. You don't actually have to go and have this archive repository.
DUSTIN SMITH: But for people in less fertile academic environments, they rely on learning tools more.
WILL FORTIN: Yeah, so I just had a conversation with a number of postdocs at Lamont about this. Because the archive is preprints, which for open access stuff is good because people put it up there and I think you said on the phone that you're supposed to be obligated to update it with the final published paper and whatever. But maybe that happens, maybe it doesn't. How many emails have I not replied to that, oof. You know?
WILL FORTIN: So I think, I don't know what it is about Earth Science, but I think we have a lot more journals. It seems that we have a way more journals and maybe just more people because Earth Science is pretty nebulous too. But, among the younger crowd, which surprised me, the postdocs I talked to in prep for this conversation, they were pretty against having something like that because it hasn't gone through peer review yet.
WILL FORTIN: So people are pretty hesitant to--
MEGAN ANSDELL: Yeah, it depends on the field. In astrophysics people rarely put stuff on the archive that hasn't been peer reviewed.
WILL FORTIN: Oh.
DUSTIN SMITH: Basically you submit to peer review submission-- you submit the manuscript and then you also post on archive?
MEGAN ANSDELL: Yeah, so some people will do that. They'll post to archive as soon as they submit something. Because like, I don't know, most-- I'm going to probably say, like I'm guessing like 80% of stuff that gets submitted for peer review will probably be published at some point. And like there's benefits to it because in my field it's pretty standard to only have one reviewer, which I think is a horrible system. And so, if you put something on the archive before when you submit it to a publisher, you get comments back from people.
MEGAN ANSDELL: You will get emails.
WILL FORTIN: You get emails.
MEGAN ANSDELL: Yeah. You can even put-- you can post it on Archive and say, this is-- I submitted this and you know comments are welcome. You see that. And so you end up getting like loads more reviews and your paper becomes better. So I think you just have to be upfront that this has not been peer reviewed. And I actually want your comments.
DUSTIN SMITH: And that's typically clear with whatever--
MEGAN ANSDELL: No. No. Yeah. I think people should do this, but they don't.
DUSTIN SMITH: Yeah.
MEGAN ANSDELL: I almost never do that just because my people who sort of raised me academically were not big believers in that. But I see it every so often. I think it's a great idea.
DUSTIN SMITH: Ingrid, do you have--
INGRID NELSON: I was just thinking that the work you guys do and the way you change data and information sounds so much purer than in medicine, where there is so much industry influence. And that's why I asked you about that. And you know, kind of always in the forefront of your mind is well, who sponsored the study? And you know who are these people.
DUSTIN SMITH: Can you trust research these days? Like what's your filtering mechanism? You go to Google Scholar and you pull a set of results.
INGRID NELSON: There are rules to assess the validity of the paper. And you know, the main journals are all peer reviewed. But this guy called-- he's a physician. He's a Greek man called Ioannidis, John Ioannidis, who published a paper in the early 2000s entitled, "Why Most Medical Research Papers Are False." And I don't know if you guys--
WILL FORTIN: No, I think a lot about that, whether most research is false, right? Because it's true until we learn the next thing, right? So I don't have too much of a problem with that. But, you know, the consequences of me locating that subduction zone wrong [INTERPOSING VOICES]
INGRID NELSON: That's kind of more the development of science. Right? So what he's saying is that these research papers are false mostly because subjects that are popular, papers about subjects that are popular tend to get published and positive results tend to get published. And so that leads to all kinds of statistical anomalies, which I didn't really understand when he described him in the paper. That you know, as a result, make these results more likely to be false than to be true.
INGRID NELSON: He wasn't quite as negative by the end of the piece. So yeah, so reading an article like that. And you know, that got a lot of-- people read that in the hospital and in medicine. And it kind of verified stuff we'd always worried about or seemed to give truth to things we'd always worried about. So it really is a huge burden. So I don't really know how you know what's true and what isn't.
DUSTIN SMITH: So beyond publication bias, I mean there are conflict of interest disclosures and things like.
INGRID NELSON: Then there's that too, yeah.
DUSTIN SMITH: Is that good enough? Does that help? Like if it says--
INGRID NELSON: That's not good enough, not when pharmaceutical companies, you know, are the main sponsors of studies. You know, it used to be you'd pick up the New England Journal of Medicine and you didn't see studies that were sponsored by drug companies. And now you always do, because that's just the-- that's who's paying for the studies.
DUSTIN SMITH: Is it a marker in your mind of quality in some way if you say, you know, study not funded by pharma and it's a significant result?
INGRID NELSON: Yeah, but I could be wrong about that. But yes.
DUSTIN SMITH: It's a thin filter, if nothing else.
INGRID NELSON: Yes, yes, yes.
WILL FORTIN: This is an Ian Bremmer can of worms, I think. Because the funding sources and where things come from can spiral so quickly.
DUSTIN SMITH: But how does that-- I mean, you're fellowship funded. But you're hawking yourself to the government.
WILL FORTIN: Oh so, yeah. My research funding comes from kind of all over. I've largely been funded by the National Science Foundation. And I've been pretty fortunate because they attach very few strings. I've also gotten money from the O&R, the Office of Naval Research, which is essentially the Department of Defense. And the Department of Energy's part of my post doc. I had some money from them.
WILL FORTIN: And so it's good to have a more diverse budgetary diet. It's just hard too.
INGRID NELSON: How much does that determine what you study, what you end up studying?
WILL FORTIN: Yeah, totally. It drives kind of everything. Because particularly things like the Department of Energy, they put out a call, where it's, we have these 12 topics that we need to go-- and so you have to take your research and cram it into one of their bins. The National Science Foundation is nice in that you don't have to do that. There are definitely-- OK, the boat is going to be in the eastern Pacific and so you better want to study something over there.
WILL FORTIN: Because if you want to go to the Indian Ocean, who's going to pay for the gas for the ship to get over there, you know? And so, you have to find a way to cram your research into a bin. I think that's true in every case. But I don't personally have felt too compromised one way or the other. You know, when you have O&R funding, yeah, sure the Navy is interested when you do research in the South China Sea.
WILL FORTIN: But, what are they doing with that research? I don't know. You know, you write it up. You put it in a journal. And they come in, they ask you questions of your post, of like what's all this turbulence happening where the bottom of submarines can dive to listed. And it's like, OK, so your submarines are diving lower and you want to know what the ocean is mixing there, cool.
WILL FORTIN: What are they doing behind the scenes? I don't know. But, for me, publishing the research ends at a journal. And I don't-- I don't-- I never had, you know, the Navy in mind when I did this research. It was you know--
DUSTIN SMITH: Are you shaping the narrative for the proposal or does it actually affect what goes all the way to publication and how you interpret results?
WILL FORTIN: I mean, you have to shape the proposal a little bit to what you think they're going to do. I've got like very few things posted up on my monitors. One of them is related to proposal writing and there's seven points. And one of them is why is this relevant to your field of research and society? And society is the part that's in italics there. And this is one of my few reminders, daily reminders, of things I'm supposed to be doing at my job.
WILL FORTIN: And you know that plays in heavily, right? You have to say to the funding body, why does this matter to people and society and the funding body itself, right? So you do tweak it a little bit for that. But I don't feel like it's the same issue that you have at all.
INGRID NELSON: I'm afraid that a lot of the influence though, is invisible. We're so used to it that we do feel comfortable with it.
WILL FORTIN: And I'm sure as I grow up and into it, I'll see more and more like, oh, no.
INGRID NELSON: I don't know, I guess.
MEGAN ANSDELL: I feel like I'm so far towards like fundamental science that like we don't have to worry about that too much. It's hard to-- well, I don't know. I guess I have had some experience in policy where you have to try to explain to people why like astrophysics research is beneficial to people in general.
DUSTIN SMITH: Just throw your phone at them.
MEGAN ANSDELL: Yeah, be like CCDs. If you have a phone in your camera, guess where that came from?
DUSTIN SMITH: Yeah.
MEGAN ANSDELL: Astronomy.
WILL FORTIN: And the networks. Yeah, I'm pretty lucky to be in Earth Science at this time of the world where people start to realize how important and fundamental that can be at large scales, climate, and smaller scales of earthquakes and landslides and so. I think there's a reason Earth Science has grown so much in the last 25 years or whenever.
DUSTIN SMITH: You guys talked about funding shaping results. Can you talk about null and significant results in both the utility side of things is the ability to publish?
WILL FORTIN: How do you mean?
DUSTIN SMITH: Like, if you have a null result.
WILL FORTIN: Oh, yeah. I just had one. Yeah I spent like a whole year working on a project, coming up with, hey, we want to use these high resolution data to look at this thing. But it turns out the high resolution data doesn't have enough offset. And so like, almost. Like I can tell you some things but not what we wanted. And it's totally disappointing because it seemed to work at first.
WILL FORTIN: And you were going down, doing this, everybody was excited about it. And then at the end you rerun a trial to compare something, and it's like, oh, I can't actually make a significant claim on this. And man, that was a no fun talk to give. Because everybody was psyched. And then it was like, yep.
DUSTIN SMITH: Did you just wait till the end and you dropped the bomb?
WILL FORTIN: No, no, no. OK, so when I say a year of my time, that's not one solid year. That's a few months over multiple years. And so I'd given talks to people about things along the way and the promising results were there and everybody was like cool, cool, cool. And then at the end when you do the final check and it doesn't work, it's disappointing. And so one of the things I wanted to do at that meeting was be like, all right, folks.
WILL FORTIN: This was a relatively small working group meeting, you know, 30 people or something. Is there any way that we could incorporate this process or what we learned or even the negative result in something that is publishable? There was just crickets in the room. And so it's going to sit unpublished.
DUSTIN SMITH: And I assume as an estimable researcher, the methods were incredibly solid and it would be a positive contribution.
WILL FORTIN: Yeah, I think it would be great. And I'm a huge fan of publishing negative results. Because some grad students going to say, hey, we have high resolution data. That's better than regular data. Let's run this process on it. It's like, dude, if it took me that long and you're a grad school just starting it's going to take you even longer. Like, this is a bad plan.
WILL FORTIN: We don't need to have somebody redo this.
MEGAN ANSDELL: But at the same time, I think it's valuable to have-- like, publish on the result and also publish your data. So like one of the things in astronomy that we have this big archive of these images of these stars and we're trying to see if we can image planets around them. And there's this huge archive that goes back like 20 years and there's all these non-detections in this. And we're astronomers. We're used to publishing non-detections.
MEGAN ANSDELL: Because we are always trying to find the faintest, smallest thing. But you know, people are now going back with these new techniques for reducing the data and pulling out smaller and smaller signals because we're getting better at things like statistics. And so, you know, publish that data. Maybe 20 years from now someone will think of some fancy technique to be able to pull out a signal.
DUSTIN SMITH: I mean your original modeling work ultimately is--
WILL FORTIN: Right. So it's not the data that's new, right? The data existed. It's the computational methods that are applied to the data to come up with detailed results. That was the process is what I was attempting and testing out there. And a fundamental limitation is in the data collection itself. So it's not quite as-- unfortunately, it's not-- I don't think it's quite analogous.
MEGAN ANSDELL: Is that not common then in your field to publish a null result?
WILL FORTIN: No. Yeah, that doesn't happen that much. [INTERPOSING VOICES]
INGRID NELSON: So, what happens to all that data?
WILL FORTIN: Yeah, it sits on my computer and dies when I get a new one. It's funny, but it sucks. It's sad.
DUSTIN SMITH: So all those postdocs would have access to your rigorous work, but they're against preprints? [LAUGHING]
MEGAN ANSDELL: Then they'd be out of a job. They're just trying-- it's just job security for them.
DUSTIN SMITH: Yeah, yeah. Ingrid. So the notion of--
INGRID NELSON: Null results?
DUSTIN SMITH: Do you have like the time and interest or do you actually just need the filter to bring you high quality, significant results?
INGRID NELSON: Well, no, null results are a big issue. Because, and I think it probably started with studies that were done looking at the use of anti-depressants, particularly in adolescents and young adults. And a lot of the data was null, especially for folks with garden variety depression. They seemed to work well with people with severe depression. But then there was this huge group of people for whom it, you know, was it the passage of time or the taking of the medicine that made a difference?
INGRID NELSON: And those results were buried. And there was a sort of a mini scandal about the use of these SSRIs in adolescents. And kind of it came to light that there were all these papers with null results that had-- studies with null results that had never been published. So I think actually that at NIH now, you can kind of go in and say, well, you know, who's doing a study on this medication?
INGRID NELSON: And who did one in the last couple of years? And what were the results? So I've actually kind of forced that data to become open and available for people. So, I mean, I guess that's sort of why a doctor would read a review article, in the hopes that we culled-- or go to a database like the Cochrane Database in the hopes that they culled all of the studies that have been done and amalgamated the results.
WILL FORTIN: Can I interrupt you for--
INGRID NELSON: Oh, sure.
WILL FORTIN: So it seems like you're saying in a review paper, there might be a sense that this null data exists.
INGRID NELSON: Yes. Who writes the review paper and how do they know--
INGRID NELSON: Yes. So that's the thing. So, yeah.
WILL FORTIN: Like, is that just conference talk and you go have a beer after, and you said, I got all this data, it's null, but, and then you have that conversation three or four times and you start going, hmm.
INGRID NELSON: Yeah, and then someone will do a literature search and--
WILL FORTIN: That seems like a terribly unstructured way to do it. I don't know. Is there a better way? I mean, I think there has to be a better way.
INGRID NELSON: I don't know. I don't know. I don't know. I don't know.
DUSTIN SMITH: Can you talk about the virtue of conferences?
WILL FORTIN: Yeah the conferences are like the best way to get things done. For me, the value of a conference is conversations like this, but without you guys, sorry. Because it's like over beers, that's where you talk about proposals. And proposals, ideas are the currency of science in my opinion. And proposals are how you trade and get-- it's the market place of that or whatever.
WILL FORTIN: And that's where you get ideas. You show new data at a conference, and sure, OK, it's important to go to the talks and see the posters and whatever. But the real value for me of a conference is in the beers or the chats or the coffees or whatever, where you say, you're doing this? I'm doing this. Cool.
WILL FORTIN: We should do the next thing. Because that's science, capital S, right? Not the journal, but the Wikipedia page. And I think that's the most valuable thing of conferences for me.
DUSTIN SMITH: Megan?
MEGAN ANSDELL: Yeah, I agree. Although I will say, I feel like, maybe it's different for you, but when I go to a conference, I'll talk to like 20 people about 20 different ideas. And like maybe one of them will eventually lead to a paper. Yeah, right?
WILL FORTIN: How many sort of proposals? We'll totally write a proposal about that.
MEGAN ANSDELL: Yeah, yeah. I'm currently about to submit a proposal that came up as an idea at a conference. I'm sitting here being like, why did I agree to do this? And you know, the observations look at taking a year from now. And maybe a year after that, we'll get the data and actually write a paper. So like these things are very, very long term. And also conferences get you excited about your science again.
MEGAN ANSDELL: You know, I spend a lot of time sitting, either sitting in a meeting or sitting in front of a computer coding. And yeah, talking to your colleagues and thinking of ideas gets you excited again.
DUSTIN SMITH: Do you go to conferences these days?
INGRID NELSON: Yeah. We have to. We have to get these things called CME points. But I agree with Megan, that it is nice to--
MEGAN ANSDELL: What are CME points?
INGRID NELSON: Continuing Medical Education.
MEGAN ANSDELL: Oh, OK. Yes.
INGRID NELSON: But I agree with Megan. It's nice to kind of talk to other people and you know, see how they're managing their huge population of uncontrolled diabetics or whatever.
MEGAN ANSDELL: Also just like, have a human connection with your colleagues. They're not just-- you're not just sending emails back and forth. Like, it makes me-- when I go to a conference and I grab a beer with a colleague and catch up with them about life in general, and I am probably more inclined to respond to their email quicker. Because I'm just you know--
INGRID NELSON: That's actually, that's kind of cool. You get a sense of what other people's lives are like in the same business.
MEGAN ANSDELL: And it's also great to have-- I feel like you are a happier person if you have a good community around you. And part of that is knowing them as people as well. And conferences are where you do that.
INGRID NELSON: Yeah, that's a good point.
DUSTIN SMITH: Talking about happiness, can you speak a little bit about--
MEGAN ANSDELL: What's that?
DUSTIN SMITH: --submission of papers and what that process is like?
MEGAN ANSDELL: Yeah. So--
DUSTIN SMITH: Tee ball! Pew! [LAUGHING]
MEGAN ANSDELL: So I just-- I'm about to submit a paper. So, it is in the Monthly Notices of the Royal Astronomical Society, which [INAUDIBLE] requests. So, the good thing is that when you get to the point of submitting a paper, you're usually so sick of that paper and your co-authors and that topic, and you just want to like, publish this thing.
MEGAN ANSDELL: So when you're sitting there and you're inputting this information to this interface and you're like, why do I have to do it? Because like all you want to do is get rid of this thing. And so it just seems, to have it like a pleasant experience is very important to our happiness, more than you think.
WILL FORTIN: And I mean, we make jokes all the time when you apply for a job or apply for colleges, people are like, this is all in the form I filled out. Why do I need to have it in a CV. Or it's all in the CV, why do I need to fill out a form? And so that there is a bit of that. And I agree. You're like, when you're ready to submit, you're ready just to be so done with that. Like, you've been hammering on it for years and you've written it and rewritten it.
WILL FORTIN: And your co-authors wanted this and that and the next thing. And it's, I mean, it's the least fun part. Like, some scientists really like writing papers I think. And they think that's the part that's fun. You know, doing the science, writing it up, and you know, whatever. I think writing the proposals is the most fun part, because that's ideas and you're doing stuff and it's cool.
DUSTIN SMITH: And there's beer.
WILL FORTIN: And there's beer. I mean there's beer and paper writing too. Maybe whiskey. But, yeah, getting it through and then having it come back and doing it again. And some papers are exciting and you're having fun doing it. Most are less fun. Because you want to be done with it. And one thing that I would like, that would be nice, is if there was some communication of expectation, of what is our timeline.
WILL FORTIN: You know that's really hard to nail down. So like I totally get that. You guys have so many reviewers not get back to you, I know. I've been that guy. But like what the input and output processes and a little bit more expectation management, right? Because if you know you're going to go to the doctor's office-- now I'm going to dig on you. And you've got to wait for an hour after you fill out your forms, then you're like, OK, I got to wait for an hour.
WILL FORTIN: But if you show up thinking, I have an appointment at 10:00 and it's 10:40, what the heck? That's a way more frustrating experience. And so, some measure of expectation and communication, would help the frustrating process. And I mean, that's impossible to nail down perfectly. But, you know.
DUSTIN SMITH: How different is the experience between all the journals you publish in?
WILL FORTIN: That's hard, right? Because I think it was you were saying you've got one or two major journals that most of your papers go to. For us, we've gotten like a dozen or more, right? And so you go to like a GRL, or you know, an AGU Journal, American Geophysical Union Journal. And the GRL is a short format, rapid response kind of thing. That can go from submission to published in like three months or less, or you know pretty quick.
WILL FORTIN: And then other ones you know, if you've got revisions and things drag on and everybody takes a long time, that can take a long time. And there's a fundamental difference between a three page letter and a 30 page, with appendix and you know 30 figures and whatever. So I mean, it takes three months just to input all those figures into the format. Because each figure is huge and terrible.
DUSTIN SMITH: How much time do you put aside to submit a paper? [CHUCKLING]
WILL FORTIN: Usually it gets done in little bits and pieces, unfortunately. I think I've only fully submitted a paper in one go like once. Yeah. So it's just mornings.
INGRID NELSON: I feel like my experience is totally different, like with these journals. Because I think it goes back to the fact that there's only three or four in astrophysics.
WILL FORTIN: So you know all the process of them.
MEGAN ANSDELL: Yeah. They're all pretty much the same. It's pretty quick. You know, as a reviewer, you're told, you know, get back to us in three or four weeks. And maybe there's been a few times where you could have to nudge them. But it's usually because their reviewer is being a pain, and not you know, anything to do with the publisher. But everything's pretty straightforward and the timeline is usually the same.
MEGAN ANSDELL: If it takes more than three months to publish from submission to it being accepted, if it takes more than three months, it's usually because your paper was bad or you have a reviewer that sucks. Sorry. That's meticulous.
INGRID NELSON: Are your guys' jobs contingent on publishing?
MEGAN ANSDELL: Yes.
WILL FORTIN: Yeah, I mean, not officially, but yeah.
DUSTIN SMITH: Can you talk about how much journal reputation matters and how you think and choose about both the reading and publishing side?
WILL FORTIN: I could talk a lot about that.
DUSTIN SMITH: I think you may have a minute.
WILL FORTIN: I think it's changing and I talked to the my division head about this yesterday, because I wanted to get his perspective over his career. And he's like, yeah, you know, there's not a pressure to go into higher journals. But there kind of is. Because once you get your name out and around and whatever, you need to please the powers that be at Columbia for you to keep your job essentially, right?
WILL FORTIN: It's the review process, tenure promotion, blah, blah, blah, whatever. And--
DUSTIN SMITH: And what do the higher journals say?
WILL FORTIN: It's easier to do if it's like Science or Nature. But that's changing. And his perspective on it was great. Because he lived through the paper version to now the digital version. He says, that's changed so much of it, because now the digital versions can be read just as widely. Because they're not physical copies sitting in the coffee room of the building or whatever. And so where the paper goes matters a lot less to us now than it used to because it's all digital.
DUSTIN SMITH: So are there two tiers? Like, is that what you're implying?
WILL FORTIN: It used to be many tiers. Like, the impact factor mattered a lot. Now it matters less. But he still is like, yeah, OK, there's the Science and Nature and then everything else. And it's like that's kind of a crude and terrible system. But you know, if Columbia comes knocking and says, did you publish your science paper? You need to have one for your tenure review. Then it's like, that's horrible.
WILL FORTIN: But that's kind of the case.
DUSTIN SMITH: Ingrid, I mean, how do you-- you have your Google Scholar, but I mean, how do you think about journal reputation and specialty journals?
INGRID NELSON: Kind of the same way Will does. Basic literacy in medicine means you read certain journals every month, every week. So you know, New England Journal, Lancet, Annals of Internal Medicine. And then specialties have their own journals. But that's just being able to have a conversation with your colleagues. What you were saying before. It's a little bit of a shame factor there too.
INGRID NELSON: But I totally agree with what Will said that, you know, now that you can Google Scholar a subject, the information much-- you don't have to go to the library and look for you know, the journal of yada, yada, yada. It's right there online. So.
DUSTIN SMITH: Yeah. Ticking through some of the-- actually landing on the journal site experiences, I mean, it's pretty generic, Google Scholar. I mean they index very well. But did any of you know sign up for any of the journals accounts and take advantage of any personalization or anything like that?
MEGAN ANSDELL: I mean I get access through my institutes usually. And I would prefer to actually get the journal copy rather than whatever is on the archive. But if I'm not connected through the VPN of the Simons Foundation, then I don't have access to that. So yeah. It's a bit of a pain, that process of like making sure that you actually have access to the journals.
MEGAN ANSDELL: So you end up going to the archive, you know, a bunch. But I would much prefer to get that the published.
INGRID NELSON: Yeah it's definitely a two-step process. So most hospitals have institutional subscriptions to almost everything.
WILL FORTIN: So on institutional subscriptions, that's, you know, another big can of worms is the fund on the journal side, right? Because we pay to have papers published. We review for free. And then the institutions pay to access the journals. And that's a hiccup for a lot of people. I've got colleagues who are at state schools who are e-mailing me all the time of like, hey, can you send me this paper?
WILL FORTIN: It's behind a paywall and they know Columbia buys everything. And that's a little bit weird. It's, you know, publicly funded stuff. And so I get there's an economic-- well, I don't get the economic model at all. I'll be honest. I get that there needs to be a economic model, right? And personally, I'd rather see everything go open access.
WILL FORTIN: And if you gotta pay to publish a paper, that's fine. Right? It gets put into a grant, so that comes out of all of your pockets and mine in tax dollars, right? Probably not maybe the best system. But it's better than having my colleagues at state schools email me, being like, hey, can you get this paper for me? Hey, can you get this paper for me? Because you're in academia?
WILL FORTIN: You're already like, if we went to industry, we'd make tons more money. We're doing this because we want to further science. And we like exploring and learning. And not everybody wants to do that. And then to have these extra hurdles put into access papers that your colleagues wrote, that's a-- that doesn't seem right.
MEGAN ANSDELL: Yeah, or your own papers. I was at a point where I was doing policy work and I needed to get a paper that I wrote. And I couldn't get access to my own paper, because I no longer had institutional access. I think I e-mailed my brother who was in med school.
DUSTIN SMITH: That's proper hustle.
MEGAN ANSDELL: Yeah.
DUSTIN SMITH: I mean, you mentioned the you know e-mailing about papers on the archive. There's some of the notion of different types of peer review. Could you imagine a scenario in which you'd allocate some time to say, you know, rate certain elements of a paper, say, veracity, you know, how much you believe in a certain result? Or give micro comments back if that were offered?
WILL FORTIN: Based on what?
DUSTIN SMITH: So imagine you're reading a paper. There was there was a figure and you thought that figure is total bullshit. So you'd give it a negative 2. And then you would put a little comment and that would go back to the editor or author.
WILL FORTIN: As a reviewer?
DUSTIN SMITH: No, just as a guy.
INGRID NELSON: I feel like that could spiral downward very quickly.
WILL FORTIN: Yeah, so, I mean, reading a paper, I don't know if you guys actually read the papers. Because I don't think we all read the papers. You read the ones that are the most relevant. You look at the figures of the other ones. And we rely on the peer review process to say whether or not something's legit. And so, yeah, you can get steamed about some figures, and, oh, man, some conference rooms have some brilliant exchanges between people who have these things.
WILL FORTIN: It's really fun when it happens. But I don't spend a lot of time going over really saying, I don't like this figure, I don't believe. I don't think in belief most of time in science, right? I trust the peer review process. We always have like two or three typically reviewers, which is nice compared to one, which, crazy.
MEGAN ANSDELL: Yeah, so, I think as a result, we have these journal clubs where people do say these things all the time. They're like, this is a horrible figure, or who let this go through the publication process? You know, this kind of thing. And so I mean, you could probably wrangle a bunch of astronomers in various departments to just type down what they talk about in their journal clubs. And this would be exactly what you're asking for.
DUSTIN SMITH: Yeah. Is there anything like journal club?
INGRID NELSON: Oh, yeah, yeah. You know, it's a big part of the-- you know, hospital I work, we have our residency programs. So that's kind of a key way to get the residents to read articles. They read them for you and learn how to analyze them.
DUSTIN SMITH: But you are having a regular exchanges with them?
INGRID NELSON: Yeah, but, and, we're probably not very critical. I've never heard anyone say, boy, this is an awful number, but it's kind of different information. But it's kind of more of an analysis. And you know, they always end up with a question mark? At least in clinical practice, should you do it or not? Well, I don't know, maybe.
DUSTIN SMITH: How do you think the peer review process is working in medicine these days?
INGRID NELSON: I totally don't know.
DUSTIN SMITH: Do you review papers.
INGRID NELSON: I have very infrequently in the past, yeah. I'm sorry. I don't know. I don't know.
DUSTIN SMITH: Perfectly acceptable answer. Well how much time do we have? Where are we?
SPEAKER 1: We have nine minutes. And we actually have some good questions that have come [INAUDIBLE]
DUSTIN SMITH: Yeah, yeah. Why don't we do that?
SPEAKER 1: We'll start off with a question from the app, which I think is everybody. What features of journalist site do you actually use beyond the downloaded [INAUDIBLE]??
WILL FORTIN: OK, so Web Science, I use their forward and backward tracking of citations a lot. That help simplify research of like, here's a cool paper that's talking about what I want, go through forwards and backwards in time. I use that. Other than that, mostly PDFs.
DUSTIN SMITH: Yeah, download the PDF, submit a paper, and then metrics, like how many people are reading. Some journals have like how many downloads this paper has had. How many people have read it. That's pretty much it.
INGRID NELSON: Pretty basic for me.
WILL FORTIN: Oh and I had a really good customer support as I said before.
AUDIENCE: Thanks for a very engaging 45 minutes. David mentioned at the beginning a paper written 10, 50 years ago.
INGRID NELSON: I think it was 2005.
AUDIENCE: Right, questioning the validity of science writing, I guess. Recently the National Academies of Sciences, Engineering, and Medicine conducted a study on the issues related to reducability and replicability and published recently a 200 page book coming out of that. Are those two concepts, reducability and replicability something that you're hearing in your spheres? And there are many, many complex issues surrounding that for both of those but just wondering if-- it certainly is a hot topic within publishing in the National Academies.
AUDIENCE: Wondering where it is at your level.
MEGAN ANSDELL: For, in astrophysics, it's huge. And a large part of that, I think, is because we have started to move towards making the data public and the codes public as well. So we have a separate sort of place where we can publish our codes. But it would be really cool to have publishing-- that you know, publish your paper. And the journals that I publish in are really good about publishing your data online, and you know, searchable databases.
MEGAN ANSDELL: But to also include code. That's, becoming more and more standard. And I mean that's key for reproducibility in my field to actually have the code that you used.
INGRID NELSON: I'm sorry. Is code data?
MEGAN ANSDELL: No. Code is the thing, like, if I have a data set, it will load the data into you know, onto your computer. And it will tell you exactly how that data was processed and the statistics you apply to it and all the manipulation that you did to get you to your final result.
WILL FORTIN: Code code is the execution of the mathematics to go from data to the result.
MEGAN ANSDELL: Yeah, yeah. And there can be a lot.
WILL FORTIN: Oh, my god. That's all that I do. It's like literally all that I do.
MEGAN ANSDELL: Right. You can hide a lot of bad things in there and you can do a lot of great things. And so to make that public is essential for reproducibility. Yeah.
WILL FORTIN: I think, in Earth Science, you know, we know that's an important thing. But I don't think folks spend too much time actually doing it. Because so much of what we do is so specific and complicated that it would, I mean, we collect data and it's a big, you know, multimillion dollar thing to go out on a ship and collect data and whatever. And people sometimes have this discussion about getting scooped by somebody else if you publish the data and make it a community data set.
WILL FORTIN: And I think that by and large role like who is going to do that? We know everyone in the field. And you have to have all these technical skills in order to even manage it. And so the idea of somebody reproducing it, it's like, oh, man it would take a lot. And so we know we should. But like there's not enough funding to have people actually do it.
WILL FORTIN: If we had more money, I think we would do it. And I think it would be a great thing. But that's an NSF budget or you know, DOE or whoever conversation. I would love to, but we just can't, because we don't have enough funding.
MEGAN ANSDELL: So NASA just started this grant I think like a few years ago. It's called Astrophysical Data Analysis Program. And it's specifically for archival data. So this is like stuff that's already been looked at or whatever. But like you're getting money just to look at archival data. And that that's helping a lot with looking at old data sets and redoing things like that.
AUDIENCE: Hi, I'm Greg Fagan from DCL. Thanks for the very interesting discussion. There's a lot of [INAUDIBLE] out there from authors that they really hate-- well maybe hate's a strong word, but strongly dislike the publication process. Different rules for submitting papers, author instructions, different peer review of systems. So where are publishers-- where can they do better to make your lives easier to get your content published?
INGRID NELSON: I don't think it's that bad.
WILL FORTIN: Well, personally, boiled frog. [LAUGHING]
WILL FORTIN: Yeah, I think you know, I think you maybe have a different system. So You've got two or three journals you go to, and you're a little more familiar. Nobody really likes making-- going through that process, I think that I know. And I think again, expectation management. Because that's something I think you guys might be able to control a little bit.
WILL FORTIN: Sure, reviewers still flake out and there's a lot of difficulties there, that it's not always an enviable job to be on your side of the chair. But it's not really fun for us either. And I think everyone up here knows and is sympathetic to that. So I'm sure you get a lot of angry things. People leave angry Google reviews or really great Google reviews, not so many in the middle.
WILL FORTIN: So you probably get a little bit of a skewed amount of hate, or too much of it. Because you know those are the squeaky wheels. But, yeah, expectation--
DUSTIN SMITH: You think the editors, they're taking your manuscript submissions in need 100% the materials in order to make the filtering decision to pass them on to peer review?
WILL FORTIN: I'm not actually sure what the editor does when you submit it to them. I never had a paper that's not gone past an editor. So I know what it would be to not go past an editor. So to me, it's like an editor gets it and then gives it to peer reviewers, that's how I see the process.
DUSTIN SMITH: I'm sure it's automatic, right. Other questions?
AUDIENCE: Michael [INAUDIBLE] from INP Consulting. Thank you. It's very interesting. A number of very interesting things. I'm interested in something Ingrid said about up-to-date and aggregation and data aggregation content aggregation is very important in this industry. It's a fairly large statement and characterization of [INAUDIBLE].
AUDIENCE: I'm just curious, if you were to go back to [INAUDIBLE] some feedback as to what they could do to kind of get that product from [INAUDIBLE] and not have to go to scholar. What are the types of things that you would instruct them or talk to you about that product that would make them more useful?
INGRID NELSON: Kind of off the top of my head, I can think of a few things. I think the search function is really a problem. I had a patient who wanted to get the morning after pill. And I couldn't remember the name of it. So I typed into up to date the morning after pill and I couldn't find it. So I went to Google. And it said, oh, it's Plan B. And you know there it was. Yeah, I knew that and everything.
INGRID NELSON: So I think there's the search function is kind of the pits. I think they break, you know, like Harrison's, you know, you kind of start with an opening statement, and then you kind of zero / and you can pick a tree to follow. Up to date, gives you this opening statement and there's a block of type, block type, block of type, block of type. It's very hard to kind of move through an article about a subject and get to-- it takes a long time to get to where you want to get to now.
INGRID NELSON: And I think the writers just throw in too much of their own opinions these days. And so there's a lot of gobbledygook and you know, here's what we think. And here's the best way to do this. And suppose you had a patient like this. Well, you know, I don't care. I've got, you know, five minutes. I've got to find the answer to what the morning after pill is called.
INGRID NELSON: And so I think those are kind of the three things that really come to mind.
AUDIENCE: [INAUDIBLE] with the Journal of the American Medical Association, just wondering, in terms of your access for content, all things being equal, if you don't have an open access mandate with your funding or in your institution, with the accessibility of your content, once it's published, play a part in where you decide to submit your paper?
WILL FORTIN: 100%. Yeah.
INGRID NELSON: It's a great question.
WILL FORTIN: Open access every time.
INGRID NELSON: That's a great question.
WILL FORTIN: Yeah, I paid to have my dissertation put open as well out of my own pocket, like 100%.
SPEAKER 1: Our last question.
AUDIENCE: This is really fascinating. I'm also dealing with Duke University Press and I'm wondering, is there a service that you wish journals were providing to you through the publication process, either before, during, or after?
MEGAN ANSDELL: I kind of wish there was more of a dialogue during the review process. Like I feel like I'm given a paper and then I give the review back. And then maybe I get a response. And like that's it. And there's very little room for discussion.
WILL FORTIN: I published a paper in a European geophysical union publication and there alongside the paper was the review with it. So the reviewers comments and my responses that went in through-- it was a little bit more communication and it got published. And then there was a second round that didn't get published along with it, which was kind of tragic in my opinion.
WILL FORTIN: But I thought that was awesome. I thought that was really cool.
MEGAN ANSDELL: And that's I mean that's been talked about a bit like in academia, like do you publish the review as well? And I definitely say yes. And I know a lot of people would definitely say no. Because they don't want to see the sausage making process, and people are embarrassed, or whatever. But you know sometimes you see papers that you're like, how did this get published? And I'm sure if you saw the review, it would be like, good job, you know there's a typo on line three or something like that.
MEGAN ANSDELL: So. It would encourage people to be better reviewers, I think.
WILL FORTIN: That happens. You know, reviewer one is always like this is great. Reviewer two is always like, yeah, OK. A couple minor changes. And reviewer three, is like, this is garbage, burn it. And it's like, so then the editor has to make a choice, I guess. And so did they go two out of three or two? And so yeah, I could see why you wouldn't want reviewer three's comments to be out there.
WILL FORTIN: But you know, there's also reviewer one.
DUSTIN SMITH: All right. That's it? Sorry.
SPEAKER 1: Sorry, guys. [APPLAUSE] Let's take a real quick selfie. Yeah.
Segment:0 .
DUSTIN SMITH: Really supposed to be like a coffee hour. Like, you basically just are awkwardly impinging upon a conversation between the core people. We're glad to have you, though. And so we're just going to talk and tell the stories. I'm going to ask some questions. But first of all, not everybody here knows each other. So they're going to introduce themselves to each other. And then we're just going to ask some questions and take it from there.
WILL FORTIN: Cool. I'm going to break the fourth wall a little bit to say hi and sorry for the people who are behind me. But yeah, I'm Will. I think I've talked to some of you on the phone and some of you not. And I'm a research scientist at Columbia University. I work at the Lamont Doherty Earth Observatory. So I'm a geophysicist doing active source marine seismology.
WILL FORTIN: So I end up writing papers, reviewing papers, reading papers, interacting with students, and postdocs, and other researchers, and you know, do the whole user side of where the sausage is made.
INGRID NELSON: Nice to meet you, Will. I'm Ingrid Nelson. I'm a doctor. I work in the Bronx at a city hospital. Pure consumer research, very occasional producer. And I'm always looking for better ways to figure things out.
DUSTIN SMITH: And what field-- what--
INGRID NELSON: So I'm an internist, and I also am a hospice and palliative care physician. So I work both outpatient and inpatient. And again, it's a city hospital I work at.
DUSTIN SMITH: What's the vibe at the city hospital, for people who don't know?
INGRID NELSON: So New York City actually has the biggest city public health system in the country. We have, I think, more than a dozen city hospitals. All comers are welcome. You know, if people can't pay their bills, that's just fine. No one gets turned away. And in the whole-- all the five boroughs, we've cobbled together a pretty good health care system. You may not get your cath in the Bronx, but you can get it at Bellevue.
INGRID NELSON: And we have transportation. So it's a very-- the population is like New York City. It's absolutely varied. And you know, it's like working in the United Nations, we say, sometimes, which is meeting now. And we're really working every day with folks who are struggling to make ends meet, are struggling to stay in the country, are struggling to stay out of jail sometimes.
INGRID NELSON: So that's the vibe.
MEGAN ANSDELL: So we have a very varied group. So I'm Megan. So I'm a research fellow at the Flatiron Institute. So I'm a computational astrophysicist. And the Flatiron Institute is sort of the in-house research arm for the Simons Foundation. And we focus on-- so the Simons Foundation is a private foundation that funds a variety of science topics, and they focus a lot on computational science. So I publish a lot.
MEGAN ANSDELL: I review papers in astrophysics. But I also interact a lot with computer scientists and machine learning people who have a very different way of publishing and sharing their work. I also have a background in science policy. So I care a lot about topics on how to engage people.
WILL FORTIN: So do you have field expeditions then? Because one of the--
DUSTIN SMITH: Into space?
WILL FORTIN: No. I mean, going to a telescope-- one of the topics in the previous session was data management and how we use that. And I think that is related to publishing, because we need to figure out what we're going to do with that and how we're going to share it. Because I go out to sea for a month or two months at a time. And it's in the North Pacific. And it's terrible, and it smells like diesel.
WILL FORTIN: But you come back with huge amounts of data. And we've got to deal with that. And we deal with it some in-house, some out. And you guys have huge data sets, I'm sure. But do you guys produce data sets if it's a largely computational facilities, or models and--
MEGAN ANSDELL: Yes.
WILL FORTIN: I mean, those are data sets. But--
MEGAN ANSDELL: So at least in our field-- I think astronomy is kind of interesting because we do-- like, we go to the telescope and we get our own personal data. You know, we've applied for time at this telescope we go. We get the data. And for a long time people kept that data on whatever format they wanted to. This data would be lost, you know, and it wouldn't be shared. But because we also have this side, where we use a lot of data from spacecraft that are funded by NASA or something like that-- and so they have to be public.
MEGAN ANSDELL: So our field has opened up a lot in that sense. So we still have ground-based observatories, where, like, an individual PI will go and gather data. But it's becoming more and more popular to have that data automatically, after, like, a year, become public in some. So we're much more organized than, I think, than other sciences. And then the Simons Foundation is very into providing big data sets that can be used for computational science and making that publicly available.
MEGAN ANSDELL: So at least where I work, it's very much encouraged to make these curated data sets and make them available to people so that everybody can use them.
DUSTIN SMITH: How do you use the data, take it all the way through publishing? Is that still a big inhibition?
WILL FORTIN: Well, yes. I mean, when you think about research-- and you know, I talk to my nieces, and nephews, and sisters, and whatever-- they think I'm-- movies have skewed it so much. They think you're having eureka moments, and you're typing on keyboards, and there's music playing. 90% of it is just managing data and, like, beating it into form where you can get it into this program or this code I wrote.
WILL FORTIN: And I work a lot in advanced computational techniques, and machine learning, and seismic inversion. So these things run on, well, supercomputers, right, or clusters of computers. And it takes 1,100 computers going for a week to come up with one result. And so we got to be really careful with our data. So we handle it so much and so carefully that it's problematic.
WILL FORTIN: Some of it's old, some of it's new. You know, you try to amass geophysical borehole data that's been collected from 1950. Nobody has the same format. So I was digging around warehouses in Cape Cod, literally dust in my nose, trying to come up with paper records of seismic data. And you can't treat that the same as modern data, you know? It's a huge problem.
WILL FORTIN: And consolidating it would be so fantastic, particularly in the face of machine learning being the future of so much of science that you need to be able to give the computers data in a format that it can deal with repeatedly, AKA no paper records, no difference between a ship from 1970 to 2012, so whatever. That's a huge problem. And it's going to take a lot of time and effort on people's part in order to get it there, in order for us to do stuff.
WILL FORTIN: And so it's really cool that you guys have standardized stuff. And I assume the hospitals are doing similar form entries and compiling records.
INGRID NELSON: Well, I'd say so, but for different reasons.
WILL FORTIN: More important.
INGRID NELSON: Yeah, well, you know, it's so we can measure outcomes and we can keep track of how much money we're losing. Because we always lose money. But medicine is funny. Because we-- well, it's not funny. Medicine-- we rely on data. And that's why you read journal articles to see what works and what doesn't work. But ultimately, it's the interaction with one person that is really at the heart of your job.
INGRID NELSON: So it's very different. It's very different. We, you know, adopt standards of care which are based on data. But our work is with one person at a time.
DUSTIN SMITH: Ingrid, can you talk more about what keeping up in your field means? And you talk about one person. Is it basically you're sort of cultivating your knowledge base at the margins, based upon your patient load for the day?
INGRID NELSON: Oh, that's, like, the dirtiest word in medicine, the N of 1, right? So we try to avoid that. We try to avoid that. So I think the big leveler in terms of-- well, in the olden days, when I went to medical school in the 1990s, everyone had Harrison's Textbook of Internal Medicine, which is about the size of an OED.
DUSTIN SMITH: [INAUDIBLE] somewhere? There we go.
INGRID NELSON: Yeah, there we go. [INTERPOSING VOICES]
INGRID NELSON: Which print is too small for me to read anymore. But it was great because-- because you looked something up. You look something up.
DUSTIN SMITH: [INAUDIBLE] the new edition right now.
INGRID NELSON: We'll talk later. So you'd look something up, and there'd be 10 citations. And two and a half hours later, you're sort of through the pathophysiology. And then you find out what probably you should be doing. And then up to date came out. And that was, like, a game-changer. And I think it had a lot of effects.
DUSTIN SMITH: Do you guys know what up to date?
INGRID NELSON: You guess know what Up to Date is? So it's a computer program, where it's a big bank of information and all different subjects in medicine written by specialists in the field with a wonderful bibliography. And so you can just go in and type out adrenal adenoma. And you used to get a very clear kind of short description and an idea of where to go next with it.
WILL FORTIN: So Wikipedia is not good enough, or WebMD?
INGRID NELSON: Well, I'm going back to Google. Going back to Google. So in standardized medicine, it made our life a lot easier. And so you could spend the five minutes you needed to figure out what to do, and then spend the rest of the two and a half minutes you're allotted with each patient actually talking to them. So that's a joke. We actually get more.
DUSTIN SMITH: [INAUDIBLE] Yeah. Go ahead.
INGRID NELSON: So yeah. So we used Up to Date for a long time. But it kind of has been a victim of its own success. I think it's gotten really much broader, and much flatter, and much harder to find things in. So funnily enough, me and many of my colleagues are kind of going back to reading articles in journals. And we find them in the bibliography of Up to Date or in Google Scholar.
DUSTIN SMITH: So instead of Up to Date, are you actually going to the Google Scholar interface and--
INGRID NELSON: Absolutely. Absolutely. Well, the bibliography of Up to Date, but then Google Scholar. Yeah.
DUSTIN SMITH: Interesting. So I mean, when do you guys think about keeping up with your field, what drives that, and how do you actually try to achieve that? I assume it's pretty futile?
WILL FORTIN: No. I probably don't read as much as I should. But I think everyone would tell you that. So I think what you should do is nebulous and false. But the way I go about it is I've got a number of papers that I think are key papers for things that my colleagues have written or I've written. And if somebody references those, it's probably relevant. And so you can skim that. So it's either a Google alert or a Web Science is typically what I use.
WILL FORTIN: So if anybody ever cites this, it pops up in my email every week of like, here's a list of papers that cited this paper, that paper, whatever. And it's pretty easy. That's the tool that I use the most-- Web Science-- to find papers.
INGRID NELSON: Who sponsors Web Science?
WILL FORTIN: Oh, geez, I should know that.
INGRID NELSON: Oh, sorry.
WILL FORTIN: Probably everybody in the room but me knows that.
DUSTIN SMITH: Hi, Chris. [INAUDIBLE]
WILL FORTIN: OK, that was my guess.
DUSTIN SMITH: The bearded gentleman over there.
WILL FORTIN: Yeah.
DUSTIN SMITH: You can--
WILL FORTIN: Had a very good interaction with your online help people the other day when I lost roaming privileges. Thank you.
INGRID NELSON: So I guess what I was asking is that really sort of unbiased and open and--
WILL FORTIN: Well, I think it seems to be. I mean, I think most people I know in our field use that. And I know you guys use something different in astro, typically. And we can talk about my reaction to that later. But it's a little slower because it takes a little while for the system, I think, to hit that this has been referenced, and it's all after peer review. And you know, most of the most pressing stuff that I need to know and what's going on here and there happens in meetings and in communities.
WILL FORTIN: And I'm lucky to be at Lamont. It's like this huge-- for the people who don't know that it's a really big Earth Science Institute at Columbia. It's 600 people. And we have so many seminars you couldn't go to all of them. And so you sit in rooms with people. And you know who's collecting what data, who's doing what, and whatever.
WILL FORTIN: And if you have a question, it's pretty easy to get a hold of somebody. So all the most cutting edge stuff is usually done that way. But I think when I was in grad school at the University of Wyoming-- it's a good geology department. A lot of rocks in Wyoming. But you don't get daily seminars there. Because who's going to go to Laramie?
WILL FORTIN: So it's nice in that way. And Web Science-- yeah, it was perfectly adequate for that. adequate for that, but [INAUDIBLE]..
MEGAN ANSDELL: To your point about saying, you know, it depends on what Institute you're at. So I did my PhD at the University of Hawaii. And although it's a beautiful place, it is like one of the most remote inhabited islands in the world. So you know we don't get a lot of people coming through.
WILL FORTIN: It's like a $600 minimum flight. Like, you're trying to get a Tuesday speaker, come on.
MEGAN ANSDELL: Yeah, not going happen. So we use the archive a lot, which is a place for preprints that focuses a lot on physics, computational science, computer science, things like that. But you know, every day you get all the papers that were published that day and you get a title and an abstract. And your sort of motivation to look at this every day is basically built on a system of like shaming each other. So people would be like, oh, did you see this paper on the archive?
MEGAN ANSDELL: And if you say no, then they're like, oh, you know. It can be academics are cut throat, especially astrophysicists.
DUSTIN SMITH: We're monsters.
MEGAN ANSDELL: Yeah. So you know, so there's journal clubs also at these institutes. So I ran a journal club when I was in grad school. And you know it happened twice a week. And you talk about whatever papers were there that day that showed up on the archive. And you can filter your archive search to only give you-- so I study protoplanetary disks. And I study planet formation. And so mine was heavily biased towards like that.
DUSTIN SMITH: Do you feel that's good enough? Is that too crude?
MEGAN ANSDELL: You know, you get-- you could-- I mean, you can obviously scan titles and if a title has catches my interest, then I'll read the abstract or something like that. But I rarely-- I'll download a bunch of papers that will sit on my desktop and I won't read them. And then I'll feel bad about deleting all of them like two weeks later when they're no longer relevant. And this cycle just perpetuates itself.
WILL FORTIN: For me it's so many tabs in Chrome.
MEGAN ANSDELL: And you never want to close your browser.
DUSTIN SMITH: When you have 64 gig of RAM. That's a big problem.
MEGAN ANSDELL: But I'm at an Institute now where people are coming through all the time, you know, and so, I'm actually getting all my news from-- I barely read the archive anymore. Because I'm just in group meetings all the time with different people. And you just hear about these things naturally. You don't actually have to go and have this archive repository.
DUSTIN SMITH: But for people in less fertile academic environments, they rely on learning tools more.
WILL FORTIN: Yeah, so I just had a conversation with a number of postdocs at Lamont about this. Because the archive is preprints, which for open access stuff is good because people put it up there and I think you said on the phone that you're supposed to be obligated to update it with the final published paper and whatever. But maybe that happens, maybe it doesn't. How many emails have I not replied to that, oof. You know?
WILL FORTIN: So I think, I don't know what it is about Earth Science, but I think we have a lot more journals. It seems that we have a way more journals and maybe just more people because Earth Science is pretty nebulous too. But, among the younger crowd, which surprised me, the postdocs I talked to in prep for this conversation, they were pretty against having something like that because it hasn't gone through peer review yet.
WILL FORTIN: So people are pretty hesitant to--
MEGAN ANSDELL: Yeah, it depends on the field. In astrophysics people rarely put stuff on the archive that hasn't been peer reviewed.
WILL FORTIN: Oh.
DUSTIN SMITH: Basically you submit to peer review submission-- you submit the manuscript and then you also post on archive?
MEGAN ANSDELL: Yeah, so some people will do that. They'll post to archive as soon as they submit something. Because like, I don't know, most-- I'm going to probably say, like I'm guessing like 80% of stuff that gets submitted for peer review will probably be published at some point. And like there's benefits to it because in my field it's pretty standard to only have one reviewer, which I think is a horrible system. And so, if you put something on the archive before when you submit it to a publisher, you get comments back from people.
MEGAN ANSDELL: You will get emails.
WILL FORTIN: You get emails.
MEGAN ANSDELL: Yeah. You can even put-- you can post it on Archive and say, this is-- I submitted this and you know comments are welcome. You see that. And so you end up getting like loads more reviews and your paper becomes better. So I think you just have to be upfront that this has not been peer reviewed. And I actually want your comments.
DUSTIN SMITH: And that's typically clear with whatever--
MEGAN ANSDELL: No. No. Yeah. I think people should do this, but they don't.
DUSTIN SMITH: Yeah.
MEGAN ANSDELL: I almost never do that just because my people who sort of raised me academically were not big believers in that. But I see it every so often. I think it's a great idea.
DUSTIN SMITH: Ingrid, do you have--
INGRID NELSON: I was just thinking that the work you guys do and the way you change data and information sounds so much purer than in medicine, where there is so much industry influence. And that's why I asked you about that. And you know, kind of always in the forefront of your mind is well, who sponsored the study? And you know who are these people.
DUSTIN SMITH: Can you trust research these days? Like what's your filtering mechanism? You go to Google Scholar and you pull a set of results.
INGRID NELSON: There are rules to assess the validity of the paper. And you know, the main journals are all peer reviewed. But this guy called-- he's a physician. He's a Greek man called Ioannidis, John Ioannidis, who published a paper in the early 2000s entitled, "Why Most Medical Research Papers Are False." And I don't know if you guys--
WILL FORTIN: No, I think a lot about that, whether most research is false, right? Because it's true until we learn the next thing, right? So I don't have too much of a problem with that. But, you know, the consequences of me locating that subduction zone wrong [INTERPOSING VOICES]
INGRID NELSON: That's kind of more the development of science. Right? So what he's saying is that these research papers are false mostly because subjects that are popular, papers about subjects that are popular tend to get published and positive results tend to get published. And so that leads to all kinds of statistical anomalies, which I didn't really understand when he described him in the paper. That you know, as a result, make these results more likely to be false than to be true.
INGRID NELSON: He wasn't quite as negative by the end of the piece. So yeah, so reading an article like that. And you know, that got a lot of-- people read that in the hospital and in medicine. And it kind of verified stuff we'd always worried about or seemed to give truth to things we'd always worried about. So it really is a huge burden. So I don't really know how you know what's true and what isn't.
DUSTIN SMITH: So beyond publication bias, I mean there are conflict of interest disclosures and things like.
INGRID NELSON: Then there's that too, yeah.
DUSTIN SMITH: Is that good enough? Does that help? Like if it says--
INGRID NELSON: That's not good enough, not when pharmaceutical companies, you know, are the main sponsors of studies. You know, it used to be you'd pick up the New England Journal of Medicine and you didn't see studies that were sponsored by drug companies. And now you always do, because that's just the-- that's who's paying for the studies.
DUSTIN SMITH: Is it a marker in your mind of quality in some way if you say, you know, study not funded by pharma and it's a significant result?
INGRID NELSON: Yeah, but I could be wrong about that. But yes.
DUSTIN SMITH: It's a thin filter, if nothing else.
INGRID NELSON: Yes, yes, yes.
WILL FORTIN: This is an Ian Bremmer can of worms, I think. Because the funding sources and where things come from can spiral so quickly.
DUSTIN SMITH: But how does that-- I mean, you're fellowship funded. But you're hawking yourself to the government.
WILL FORTIN: Oh so, yeah. My research funding comes from kind of all over. I've largely been funded by the National Science Foundation. And I've been pretty fortunate because they attach very few strings. I've also gotten money from the O&R, the Office of Naval Research, which is essentially the Department of Defense. And the Department of Energy's part of my post doc. I had some money from them.
WILL FORTIN: And so it's good to have a more diverse budgetary diet. It's just hard too.
INGRID NELSON: How much does that determine what you study, what you end up studying?
WILL FORTIN: Yeah, totally. It drives kind of everything. Because particularly things like the Department of Energy, they put out a call, where it's, we have these 12 topics that we need to go-- and so you have to take your research and cram it into one of their bins. The National Science Foundation is nice in that you don't have to do that. There are definitely-- OK, the boat is going to be in the eastern Pacific and so you better want to study something over there.
WILL FORTIN: Because if you want to go to the Indian Ocean, who's going to pay for the gas for the ship to get over there, you know? And so, you have to find a way to cram your research into a bin. I think that's true in every case. But I don't personally have felt too compromised one way or the other. You know, when you have O&R funding, yeah, sure the Navy is interested when you do research in the South China Sea.
WILL FORTIN: But, what are they doing with that research? I don't know. You know, you write it up. You put it in a journal. And they come in, they ask you questions of your post, of like what's all this turbulence happening where the bottom of submarines can dive to listed. And it's like, OK, so your submarines are diving lower and you want to know what the ocean is mixing there, cool.
WILL FORTIN: What are they doing behind the scenes? I don't know. But, for me, publishing the research ends at a journal. And I don't-- I don't-- I never had, you know, the Navy in mind when I did this research. It was you know--
DUSTIN SMITH: Are you shaping the narrative for the proposal or does it actually affect what goes all the way to publication and how you interpret results?
WILL FORTIN: I mean, you have to shape the proposal a little bit to what you think they're going to do. I've got like very few things posted up on my monitors. One of them is related to proposal writing and there's seven points. And one of them is why is this relevant to your field of research and society? And society is the part that's in italics there. And this is one of my few reminders, daily reminders, of things I'm supposed to be doing at my job.
WILL FORTIN: And you know that plays in heavily, right? You have to say to the funding body, why does this matter to people and society and the funding body itself, right? So you do tweak it a little bit for that. But I don't feel like it's the same issue that you have at all.
INGRID NELSON: I'm afraid that a lot of the influence though, is invisible. We're so used to it that we do feel comfortable with it.
WILL FORTIN: And I'm sure as I grow up and into it, I'll see more and more like, oh, no.
INGRID NELSON: I don't know, I guess.
MEGAN ANSDELL: I feel like I'm so far towards like fundamental science that like we don't have to worry about that too much. It's hard to-- well, I don't know. I guess I have had some experience in policy where you have to try to explain to people why like astrophysics research is beneficial to people in general.
DUSTIN SMITH: Just throw your phone at them.
MEGAN ANSDELL: Yeah, be like CCDs. If you have a phone in your camera, guess where that came from?
DUSTIN SMITH: Yeah.
MEGAN ANSDELL: Astronomy.
WILL FORTIN: And the networks. Yeah, I'm pretty lucky to be in Earth Science at this time of the world where people start to realize how important and fundamental that can be at large scales, climate, and smaller scales of earthquakes and landslides and so. I think there's a reason Earth Science has grown so much in the last 25 years or whenever.
DUSTIN SMITH: You guys talked about funding shaping results. Can you talk about null and significant results in both the utility side of things is the ability to publish?
WILL FORTIN: How do you mean?
DUSTIN SMITH: Like, if you have a null result.
WILL FORTIN: Oh, yeah. I just had one. Yeah I spent like a whole year working on a project, coming up with, hey, we want to use these high resolution data to look at this thing. But it turns out the high resolution data doesn't have enough offset. And so like, almost. Like I can tell you some things but not what we wanted. And it's totally disappointing because it seemed to work at first.
WILL FORTIN: And you were going down, doing this, everybody was excited about it. And then at the end you rerun a trial to compare something, and it's like, oh, I can't actually make a significant claim on this. And man, that was a no fun talk to give. Because everybody was psyched. And then it was like, yep.
DUSTIN SMITH: Did you just wait till the end and you dropped the bomb?
WILL FORTIN: No, no, no. OK, so when I say a year of my time, that's not one solid year. That's a few months over multiple years. And so I'd given talks to people about things along the way and the promising results were there and everybody was like cool, cool, cool. And then at the end when you do the final check and it doesn't work, it's disappointing. And so one of the things I wanted to do at that meeting was be like, all right, folks.
WILL FORTIN: This was a relatively small working group meeting, you know, 30 people or something. Is there any way that we could incorporate this process or what we learned or even the negative result in something that is publishable? There was just crickets in the room. And so it's going to sit unpublished.
DUSTIN SMITH: And I assume as an estimable researcher, the methods were incredibly solid and it would be a positive contribution.
WILL FORTIN: Yeah, I think it would be great. And I'm a huge fan of publishing negative results. Because some grad students going to say, hey, we have high resolution data. That's better than regular data. Let's run this process on it. It's like, dude, if it took me that long and you're a grad school just starting it's going to take you even longer. Like, this is a bad plan.
WILL FORTIN: We don't need to have somebody redo this.
MEGAN ANSDELL: But at the same time, I think it's valuable to have-- like, publish on the result and also publish your data. So like one of the things in astronomy that we have this big archive of these images of these stars and we're trying to see if we can image planets around them. And there's this huge archive that goes back like 20 years and there's all these non-detections in this. And we're astronomers. We're used to publishing non-detections.
MEGAN ANSDELL: Because we are always trying to find the faintest, smallest thing. But you know, people are now going back with these new techniques for reducing the data and pulling out smaller and smaller signals because we're getting better at things like statistics. And so, you know, publish that data. Maybe 20 years from now someone will think of some fancy technique to be able to pull out a signal.
DUSTIN SMITH: I mean your original modeling work ultimately is--
WILL FORTIN: Right. So it's not the data that's new, right? The data existed. It's the computational methods that are applied to the data to come up with detailed results. That was the process is what I was attempting and testing out there. And a fundamental limitation is in the data collection itself. So it's not quite as-- unfortunately, it's not-- I don't think it's quite analogous.
MEGAN ANSDELL: Is that not common then in your field to publish a null result?
WILL FORTIN: No. Yeah, that doesn't happen that much. [INTERPOSING VOICES]
INGRID NELSON: So, what happens to all that data?
WILL FORTIN: Yeah, it sits on my computer and dies when I get a new one. It's funny, but it sucks. It's sad.
DUSTIN SMITH: So all those postdocs would have access to your rigorous work, but they're against preprints? [LAUGHING]
MEGAN ANSDELL: Then they'd be out of a job. They're just trying-- it's just job security for them.
DUSTIN SMITH: Yeah, yeah. Ingrid. So the notion of--
INGRID NELSON: Null results?
DUSTIN SMITH: Do you have like the time and interest or do you actually just need the filter to bring you high quality, significant results?
INGRID NELSON: Well, no, null results are a big issue. Because, and I think it probably started with studies that were done looking at the use of anti-depressants, particularly in adolescents and young adults. And a lot of the data was null, especially for folks with garden variety depression. They seemed to work well with people with severe depression. But then there was this huge group of people for whom it, you know, was it the passage of time or the taking of the medicine that made a difference?
INGRID NELSON: And those results were buried. And there was a sort of a mini scandal about the use of these SSRIs in adolescents. And kind of it came to light that there were all these papers with null results that had-- studies with null results that had never been published. So I think actually that at NIH now, you can kind of go in and say, well, you know, who's doing a study on this medication?
INGRID NELSON: And who did one in the last couple of years? And what were the results? So I've actually kind of forced that data to become open and available for people. So, I mean, I guess that's sort of why a doctor would read a review article, in the hopes that we culled-- or go to a database like the Cochrane Database in the hopes that they culled all of the studies that have been done and amalgamated the results.
WILL FORTIN: Can I interrupt you for--
INGRID NELSON: Oh, sure.
WILL FORTIN: So it seems like you're saying in a review paper, there might be a sense that this null data exists.
INGRID NELSON: Yes. Who writes the review paper and how do they know--
INGRID NELSON: Yes. So that's the thing. So, yeah.
WILL FORTIN: Like, is that just conference talk and you go have a beer after, and you said, I got all this data, it's null, but, and then you have that conversation three or four times and you start going, hmm.
INGRID NELSON: Yeah, and then someone will do a literature search and--
WILL FORTIN: That seems like a terribly unstructured way to do it. I don't know. Is there a better way? I mean, I think there has to be a better way.
INGRID NELSON: I don't know. I don't know. I don't know. I don't know.
DUSTIN SMITH: Can you talk about the virtue of conferences?
WILL FORTIN: Yeah the conferences are like the best way to get things done. For me, the value of a conference is conversations like this, but without you guys, sorry. Because it's like over beers, that's where you talk about proposals. And proposals, ideas are the currency of science in my opinion. And proposals are how you trade and get-- it's the market place of that or whatever.
WILL FORTIN: And that's where you get ideas. You show new data at a conference, and sure, OK, it's important to go to the talks and see the posters and whatever. But the real value for me of a conference is in the beers or the chats or the coffees or whatever, where you say, you're doing this? I'm doing this. Cool.
WILL FORTIN: We should do the next thing. Because that's science, capital S, right? Not the journal, but the Wikipedia page. And I think that's the most valuable thing of conferences for me.
DUSTIN SMITH: Megan?
MEGAN ANSDELL: Yeah, I agree. Although I will say, I feel like, maybe it's different for you, but when I go to a conference, I'll talk to like 20 people about 20 different ideas. And like maybe one of them will eventually lead to a paper. Yeah, right?
WILL FORTIN: How many sort of proposals? We'll totally write a proposal about that.
MEGAN ANSDELL: Yeah, yeah. I'm currently about to submit a proposal that came up as an idea at a conference. I'm sitting here being like, why did I agree to do this? And you know, the observations look at taking a year from now. And maybe a year after that, we'll get the data and actually write a paper. So like these things are very, very long term. And also conferences get you excited about your science again.
MEGAN ANSDELL: You know, I spend a lot of time sitting, either sitting in a meeting or sitting in front of a computer coding. And yeah, talking to your colleagues and thinking of ideas gets you excited again.
DUSTIN SMITH: Do you go to conferences these days?
INGRID NELSON: Yeah. We have to. We have to get these things called CME points. But I agree with Megan, that it is nice to--
MEGAN ANSDELL: What are CME points?
INGRID NELSON: Continuing Medical Education.
MEGAN ANSDELL: Oh, OK. Yes.
INGRID NELSON: But I agree with Megan. It's nice to kind of talk to other people and you know, see how they're managing their huge population of uncontrolled diabetics or whatever.
MEGAN ANSDELL: Also just like, have a human connection with your colleagues. They're not just-- you're not just sending emails back and forth. Like, it makes me-- when I go to a conference and I grab a beer with a colleague and catch up with them about life in general, and I am probably more inclined to respond to their email quicker. Because I'm just you know--
INGRID NELSON: That's actually, that's kind of cool. You get a sense of what other people's lives are like in the same business.
MEGAN ANSDELL: And it's also great to have-- I feel like you are a happier person if you have a good community around you. And part of that is knowing them as people as well. And conferences are where you do that.
INGRID NELSON: Yeah, that's a good point.
DUSTIN SMITH: Talking about happiness, can you speak a little bit about--
MEGAN ANSDELL: What's that?
DUSTIN SMITH: --submission of papers and what that process is like?
MEGAN ANSDELL: Yeah. So--
DUSTIN SMITH: Tee ball! Pew! [LAUGHING]
MEGAN ANSDELL: So I just-- I'm about to submit a paper. So, it is in the Monthly Notices of the Royal Astronomical Society, which [INAUDIBLE] requests. So, the good thing is that when you get to the point of submitting a paper, you're usually so sick of that paper and your co-authors and that topic, and you just want to like, publish this thing.
MEGAN ANSDELL: So when you're sitting there and you're inputting this information to this interface and you're like, why do I have to do it? Because like all you want to do is get rid of this thing. And so it just seems, to have it like a pleasant experience is very important to our happiness, more than you think.
WILL FORTIN: And I mean, we make jokes all the time when you apply for a job or apply for colleges, people are like, this is all in the form I filled out. Why do I need to have it in a CV. Or it's all in the CV, why do I need to fill out a form? And so that there is a bit of that. And I agree. You're like, when you're ready to submit, you're ready just to be so done with that. Like, you've been hammering on it for years and you've written it and rewritten it.
WILL FORTIN: And your co-authors wanted this and that and the next thing. And it's, I mean, it's the least fun part. Like, some scientists really like writing papers I think. And they think that's the part that's fun. You know, doing the science, writing it up, and you know, whatever. I think writing the proposals is the most fun part, because that's ideas and you're doing stuff and it's cool.
DUSTIN SMITH: And there's beer.
WILL FORTIN: And there's beer. I mean there's beer and paper writing too. Maybe whiskey. But, yeah, getting it through and then having it come back and doing it again. And some papers are exciting and you're having fun doing it. Most are less fun. Because you want to be done with it. And one thing that I would like, that would be nice, is if there was some communication of expectation, of what is our timeline.
WILL FORTIN: You know that's really hard to nail down. So like I totally get that. You guys have so many reviewers not get back to you, I know. I've been that guy. But like what the input and output processes and a little bit more expectation management, right? Because if you know you're going to go to the doctor's office-- now I'm going to dig on you. And you've got to wait for an hour after you fill out your forms, then you're like, OK, I got to wait for an hour.
WILL FORTIN: But if you show up thinking, I have an appointment at 10:00 and it's 10:40, what the heck? That's a way more frustrating experience. And so, some measure of expectation and communication, would help the frustrating process. And I mean, that's impossible to nail down perfectly. But, you know.
DUSTIN SMITH: How different is the experience between all the journals you publish in?
WILL FORTIN: That's hard, right? Because I think it was you were saying you've got one or two major journals that most of your papers go to. For us, we've gotten like a dozen or more, right? And so you go to like a GRL, or you know, an AGU Journal, American Geophysical Union Journal. And the GRL is a short format, rapid response kind of thing. That can go from submission to published in like three months or less, or you know pretty quick.
WILL FORTIN: And then other ones you know, if you've got revisions and things drag on and everybody takes a long time, that can take a long time. And there's a fundamental difference between a three page letter and a 30 page, with appendix and you know 30 figures and whatever. So I mean, it takes three months just to input all those figures into the format. Because each figure is huge and terrible.
DUSTIN SMITH: How much time do you put aside to submit a paper? [CHUCKLING]
WILL FORTIN: Usually it gets done in little bits and pieces, unfortunately. I think I've only fully submitted a paper in one go like once. Yeah. So it's just mornings.
INGRID NELSON: I feel like my experience is totally different, like with these journals. Because I think it goes back to the fact that there's only three or four in astrophysics.
WILL FORTIN: So you know all the process of them.
MEGAN ANSDELL: Yeah. They're all pretty much the same. It's pretty quick. You know, as a reviewer, you're told, you know, get back to us in three or four weeks. And maybe there's been a few times where you could have to nudge them. But it's usually because their reviewer is being a pain, and not you know, anything to do with the publisher. But everything's pretty straightforward and the timeline is usually the same.
MEGAN ANSDELL: If it takes more than three months to publish from submission to it being accepted, if it takes more than three months, it's usually because your paper was bad or you have a reviewer that sucks. Sorry. That's meticulous.
INGRID NELSON: Are your guys' jobs contingent on publishing?
MEGAN ANSDELL: Yes.
WILL FORTIN: Yeah, I mean, not officially, but yeah.
DUSTIN SMITH: Can you talk about how much journal reputation matters and how you think and choose about both the reading and publishing side?
WILL FORTIN: I could talk a lot about that.
DUSTIN SMITH: I think you may have a minute.
WILL FORTIN: I think it's changing and I talked to the my division head about this yesterday, because I wanted to get his perspective over his career. And he's like, yeah, you know, there's not a pressure to go into higher journals. But there kind of is. Because once you get your name out and around and whatever, you need to please the powers that be at Columbia for you to keep your job essentially, right?
WILL FORTIN: It's the review process, tenure promotion, blah, blah, blah, whatever. And--
DUSTIN SMITH: And what do the higher journals say?
WILL FORTIN: It's easier to do if it's like Science or Nature. But that's changing. And his perspective on it was great. Because he lived through the paper version to now the digital version. He says, that's changed so much of it, because now the digital versions can be read just as widely. Because they're not physical copies sitting in the coffee room of the building or whatever. And so where the paper goes matters a lot less to us now than it used to because it's all digital.
DUSTIN SMITH: So are there two tiers? Like, is that what you're implying?
WILL FORTIN: It used to be many tiers. Like, the impact factor mattered a lot. Now it matters less. But he still is like, yeah, OK, there's the Science and Nature and then everything else. And it's like that's kind of a crude and terrible system. But you know, if Columbia comes knocking and says, did you publish your science paper? You need to have one for your tenure review. Then it's like, that's horrible.
WILL FORTIN: But that's kind of the case.
DUSTIN SMITH: Ingrid, I mean, how do you-- you have your Google Scholar, but I mean, how do you think about journal reputation and specialty journals?
INGRID NELSON: Kind of the same way Will does. Basic literacy in medicine means you read certain journals every month, every week. So you know, New England Journal, Lancet, Annals of Internal Medicine. And then specialties have their own journals. But that's just being able to have a conversation with your colleagues. What you were saying before. It's a little bit of a shame factor there too.
INGRID NELSON: But I totally agree with what Will said that, you know, now that you can Google Scholar a subject, the information much-- you don't have to go to the library and look for you know, the journal of yada, yada, yada. It's right there online. So.
DUSTIN SMITH: Yeah. Ticking through some of the-- actually landing on the journal site experiences, I mean, it's pretty generic, Google Scholar. I mean they index very well. But did any of you know sign up for any of the journals accounts and take advantage of any personalization or anything like that?
MEGAN ANSDELL: I mean I get access through my institutes usually. And I would prefer to actually get the journal copy rather than whatever is on the archive. But if I'm not connected through the VPN of the Simons Foundation, then I don't have access to that. So yeah. It's a bit of a pain, that process of like making sure that you actually have access to the journals.
MEGAN ANSDELL: So you end up going to the archive, you know, a bunch. But I would much prefer to get that the published.
INGRID NELSON: Yeah it's definitely a two-step process. So most hospitals have institutional subscriptions to almost everything.
WILL FORTIN: So on institutional subscriptions, that's, you know, another big can of worms is the fund on the journal side, right? Because we pay to have papers published. We review for free. And then the institutions pay to access the journals. And that's a hiccup for a lot of people. I've got colleagues who are at state schools who are e-mailing me all the time of like, hey, can you send me this paper?
WILL FORTIN: It's behind a paywall and they know Columbia buys everything. And that's a little bit weird. It's, you know, publicly funded stuff. And so I get there's an economic-- well, I don't get the economic model at all. I'll be honest. I get that there needs to be a economic model, right? And personally, I'd rather see everything go open access.
WILL FORTIN: And if you gotta pay to publish a paper, that's fine. Right? It gets put into a grant, so that comes out of all of your pockets and mine in tax dollars, right? Probably not maybe the best system. But it's better than having my colleagues at state schools email me, being like, hey, can you get this paper for me? Hey, can you get this paper for me? Because you're in academia?
WILL FORTIN: You're already like, if we went to industry, we'd make tons more money. We're doing this because we want to further science. And we like exploring and learning. And not everybody wants to do that. And then to have these extra hurdles put into access papers that your colleagues wrote, that's a-- that doesn't seem right.
MEGAN ANSDELL: Yeah, or your own papers. I was at a point where I was doing policy work and I needed to get a paper that I wrote. And I couldn't get access to my own paper, because I no longer had institutional access. I think I e-mailed my brother who was in med school.
DUSTIN SMITH: That's proper hustle.
MEGAN ANSDELL: Yeah.
DUSTIN SMITH: I mean, you mentioned the you know e-mailing about papers on the archive. There's some of the notion of different types of peer review. Could you imagine a scenario in which you'd allocate some time to say, you know, rate certain elements of a paper, say, veracity, you know, how much you believe in a certain result? Or give micro comments back if that were offered?
WILL FORTIN: Based on what?
DUSTIN SMITH: So imagine you're reading a paper. There was there was a figure and you thought that figure is total bullshit. So you'd give it a negative 2. And then you would put a little comment and that would go back to the editor or author.
WILL FORTIN: As a reviewer?
DUSTIN SMITH: No, just as a guy.
INGRID NELSON: I feel like that could spiral downward very quickly.
WILL FORTIN: Yeah, so, I mean, reading a paper, I don't know if you guys actually read the papers. Because I don't think we all read the papers. You read the ones that are the most relevant. You look at the figures of the other ones. And we rely on the peer review process to say whether or not something's legit. And so, yeah, you can get steamed about some figures, and, oh, man, some conference rooms have some brilliant exchanges between people who have these things.
WILL FORTIN: It's really fun when it happens. But I don't spend a lot of time going over really saying, I don't like this figure, I don't believe. I don't think in belief most of time in science, right? I trust the peer review process. We always have like two or three typically reviewers, which is nice compared to one, which, crazy.
MEGAN ANSDELL: Yeah, so, I think as a result, we have these journal clubs where people do say these things all the time. They're like, this is a horrible figure, or who let this go through the publication process? You know, this kind of thing. And so I mean, you could probably wrangle a bunch of astronomers in various departments to just type down what they talk about in their journal clubs. And this would be exactly what you're asking for.
DUSTIN SMITH: Yeah. Is there anything like journal club?
INGRID NELSON: Oh, yeah, yeah. You know, it's a big part of the-- you know, hospital I work, we have our residency programs. So that's kind of a key way to get the residents to read articles. They read them for you and learn how to analyze them.
DUSTIN SMITH: But you are having a regular exchanges with them?
INGRID NELSON: Yeah, but, and, we're probably not very critical. I've never heard anyone say, boy, this is an awful number, but it's kind of different information. But it's kind of more of an analysis. And you know, they always end up with a question mark? At least in clinical practice, should you do it or not? Well, I don't know, maybe.
DUSTIN SMITH: How do you think the peer review process is working in medicine these days?
INGRID NELSON: I totally don't know.
DUSTIN SMITH: Do you review papers.
INGRID NELSON: I have very infrequently in the past, yeah. I'm sorry. I don't know. I don't know.
DUSTIN SMITH: Perfectly acceptable answer. Well how much time do we have? Where are we?
SPEAKER 1: We have nine minutes. And we actually have some good questions that have come [INAUDIBLE]
DUSTIN SMITH: Yeah, yeah. Why don't we do that?
SPEAKER 1: We'll start off with a question from the app, which I think is everybody. What features of journalist site do you actually use beyond the downloaded [INAUDIBLE]??
WILL FORTIN: OK, so Web Science, I use their forward and backward tracking of citations a lot. That help simplify research of like, here's a cool paper that's talking about what I want, go through forwards and backwards in time. I use that. Other than that, mostly PDFs.
DUSTIN SMITH: Yeah, download the PDF, submit a paper, and then metrics, like how many people are reading. Some journals have like how many downloads this paper has had. How many people have read it. That's pretty much it.
INGRID NELSON: Pretty basic for me.
WILL FORTIN: Oh and I had a really good customer support as I said before.
AUDIENCE: Thanks for a very engaging 45 minutes. David mentioned at the beginning a paper written 10, 50 years ago.
INGRID NELSON: I think it was 2005.
AUDIENCE: Right, questioning the validity of science writing, I guess. Recently the National Academies of Sciences, Engineering, and Medicine conducted a study on the issues related to reducability and replicability and published recently a 200 page book coming out of that. Are those two concepts, reducability and replicability something that you're hearing in your spheres? And there are many, many complex issues surrounding that for both of those but just wondering if-- it certainly is a hot topic within publishing in the National Academies.
AUDIENCE: Wondering where it is at your level.
MEGAN ANSDELL: For, in astrophysics, it's huge. And a large part of that, I think, is because we have started to move towards making the data public and the codes public as well. So we have a separate sort of place where we can publish our codes. But it would be really cool to have publishing-- that you know, publish your paper. And the journals that I publish in are really good about publishing your data online, and you know, searchable databases.
MEGAN ANSDELL: But to also include code. That's, becoming more and more standard. And I mean that's key for reproducibility in my field to actually have the code that you used.
INGRID NELSON: I'm sorry. Is code data?
MEGAN ANSDELL: No. Code is the thing, like, if I have a data set, it will load the data into you know, onto your computer. And it will tell you exactly how that data was processed and the statistics you apply to it and all the manipulation that you did to get you to your final result.
WILL FORTIN: Code code is the execution of the mathematics to go from data to the result.
MEGAN ANSDELL: Yeah, yeah. And there can be a lot.
WILL FORTIN: Oh, my god. That's all that I do. It's like literally all that I do.
MEGAN ANSDELL: Right. You can hide a lot of bad things in there and you can do a lot of great things. And so to make that public is essential for reproducibility. Yeah.
WILL FORTIN: I think, in Earth Science, you know, we know that's an important thing. But I don't think folks spend too much time actually doing it. Because so much of what we do is so specific and complicated that it would, I mean, we collect data and it's a big, you know, multimillion dollar thing to go out on a ship and collect data and whatever. And people sometimes have this discussion about getting scooped by somebody else if you publish the data and make it a community data set.
WILL FORTIN: And I think that by and large role like who is going to do that? We know everyone in the field. And you have to have all these technical skills in order to even manage it. And so the idea of somebody reproducing it, it's like, oh, man it would take a lot. And so we know we should. But like there's not enough funding to have people actually do it.
WILL FORTIN: If we had more money, I think we would do it. And I think it would be a great thing. But that's an NSF budget or you know, DOE or whoever conversation. I would love to, but we just can't, because we don't have enough funding.
MEGAN ANSDELL: So NASA just started this grant I think like a few years ago. It's called Astrophysical Data Analysis Program. And it's specifically for archival data. So this is like stuff that's already been looked at or whatever. But like you're getting money just to look at archival data. And that that's helping a lot with looking at old data sets and redoing things like that.
AUDIENCE: Hi, I'm Greg Fagan from DCL. Thanks for the very interesting discussion. There's a lot of [INAUDIBLE] out there from authors that they really hate-- well maybe hate's a strong word, but strongly dislike the publication process. Different rules for submitting papers, author instructions, different peer review of systems. So where are publishers-- where can they do better to make your lives easier to get your content published?
INGRID NELSON: I don't think it's that bad.
WILL FORTIN: Well, personally, boiled frog. [LAUGHING]
WILL FORTIN: Yeah, I think you know, I think you maybe have a different system. So You've got two or three journals you go to, and you're a little more familiar. Nobody really likes making-- going through that process, I think that I know. And I think again, expectation management. Because that's something I think you guys might be able to control a little bit.
WILL FORTIN: Sure, reviewers still flake out and there's a lot of difficulties there, that it's not always an enviable job to be on your side of the chair. But it's not really fun for us either. And I think everyone up here knows and is sympathetic to that. So I'm sure you get a lot of angry things. People leave angry Google reviews or really great Google reviews, not so many in the middle.
WILL FORTIN: So you probably get a little bit of a skewed amount of hate, or too much of it. Because you know those are the squeaky wheels. But, yeah, expectation--
DUSTIN SMITH: You think the editors, they're taking your manuscript submissions in need 100% the materials in order to make the filtering decision to pass them on to peer review?
WILL FORTIN: I'm not actually sure what the editor does when you submit it to them. I never had a paper that's not gone past an editor. So I know what it would be to not go past an editor. So to me, it's like an editor gets it and then gives it to peer reviewers, that's how I see the process.
DUSTIN SMITH: I'm sure it's automatic, right. Other questions?
AUDIENCE: Michael [INAUDIBLE] from INP Consulting. Thank you. It's very interesting. A number of very interesting things. I'm interested in something Ingrid said about up-to-date and aggregation and data aggregation content aggregation is very important in this industry. It's a fairly large statement and characterization of [INAUDIBLE].
AUDIENCE: I'm just curious, if you were to go back to [INAUDIBLE] some feedback as to what they could do to kind of get that product from [INAUDIBLE] and not have to go to scholar. What are the types of things that you would instruct them or talk to you about that product that would make them more useful?
INGRID NELSON: Kind of off the top of my head, I can think of a few things. I think the search function is really a problem. I had a patient who wanted to get the morning after pill. And I couldn't remember the name of it. So I typed into up to date the morning after pill and I couldn't find it. So I went to Google. And it said, oh, it's Plan B. And you know there it was. Yeah, I knew that and everything.
INGRID NELSON: So I think there's the search function is kind of the pits. I think they break, you know, like Harrison's, you know, you kind of start with an opening statement, and then you kind of zero / and you can pick a tree to follow. Up to date, gives you this opening statement and there's a block of type, block type, block of type, block of type. It's very hard to kind of move through an article about a subject and get to-- it takes a long time to get to where you want to get to now.
INGRID NELSON: And I think the writers just throw in too much of their own opinions these days. And so there's a lot of gobbledygook and you know, here's what we think. And here's the best way to do this. And suppose you had a patient like this. Well, you know, I don't care. I've got, you know, five minutes. I've got to find the answer to what the morning after pill is called.
INGRID NELSON: And so I think those are kind of the three things that really come to mind.
AUDIENCE: [INAUDIBLE] with the Journal of the American Medical Association, just wondering, in terms of your access for content, all things being equal, if you don't have an open access mandate with your funding or in your institution, with the accessibility of your content, once it's published, play a part in where you decide to submit your paper?
WILL FORTIN: 100%. Yeah.
INGRID NELSON: It's a great question.
WILL FORTIN: Open access every time.
INGRID NELSON: That's a great question.
WILL FORTIN: Yeah, I paid to have my dissertation put open as well out of my own pocket, like 100%.
SPEAKER 1: Our last question.
AUDIENCE: This is really fascinating. I'm also dealing with Duke University Press and I'm wondering, is there a service that you wish journals were providing to you through the publication process, either before, during, or after?
MEGAN ANSDELL: I kind of wish there was more of a dialogue during the review process. Like I feel like I'm given a paper and then I give the review back. And then maybe I get a response. And like that's it. And there's very little room for discussion.
WILL FORTIN: I published a paper in a European geophysical union publication and there alongside the paper was the review with it. So the reviewers comments and my responses that went in through-- it was a little bit more communication and it got published. And then there was a second round that didn't get published along with it, which was kind of tragic in my opinion.
WILL FORTIN: But I thought that was awesome. I thought that was really cool.
MEGAN ANSDELL: And that's I mean that's been talked about a bit like in academia, like do you publish the review as well? And I definitely say yes. And I know a lot of people would definitely say no. Because they don't want to see the sausage making process, and people are embarrassed, or whatever. But you know sometimes you see papers that you're like, how did this get published? And I'm sure if you saw the review, it would be like, good job, you know there's a typo on line three or something like that.
MEGAN ANSDELL: So. It would encourage people to be better reviewers, I think.
WILL FORTIN: That happens. You know, reviewer one is always like this is great. Reviewer two is always like, yeah, OK. A couple minor changes. And reviewer three, is like, this is garbage, burn it. And it's like, so then the editor has to make a choice, I guess. And so did they go two out of three or two? And so yeah, I could see why you wouldn't want reviewer three's comments to be out there.
WILL FORTIN: But you know, there's also reviewer one.
DUSTIN SMITH: All right. That's it? Sorry.
SPEAKER 1: Sorry, guys. [APPLAUSE] Let's take a real quick selfie. Yeah.
Form Title Download
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility and Keyboard Shortcuts
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html video player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the video player are generally tabs, one of which shows a scrolling transcript of the video and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the video is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the video by opening up an explore dialog which contains tabs for a visual navigation of the video using thumbnails, an about tab which contains rich metadata about the media, a segments tab which contains further information about the video, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
I: Info Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus / Exits Player
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Exit Clip Mode
You have requested to see a portion of the video outside of the clip.
Click OK to switch to the full video or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode