[MUSIC] Well, here we are in the very first lecture, thanks for joining me. And before we get into the nitty-gritty of how to conduct a medical study or how to interpret a medical study we're going to talk in a very broad sense about how medical research works in general, the 10,000-foot view as we say. Here are the goals of this lecture. We're going to describe the process of medical research. How we get from A to Z from a hypothesis to a manuscript. We're going to talk about the concept of peer review. We're going to explain what open data means and what its promise is and how that promise sometimes gets broken. And finally, we're going to talk about some of the perverse incentives that underlie all of medical science. But first I want to tell you about the greatest salesman in the world. Imagine you're in the market for a new food processor [SOUND]. [MUSIC] You decide to go to the number one food processor store in the world the New England Food Processor Emporium. Inside a Salesman shows you a beautiful box. There's a picture on the front a lovely food processor with all the bells and whistles. There are multiple testimonials. Best food processor I've ever used. Makes other food processors look like a garbage disposal. Saved my waistline and my marriage. Very impressive stuff. You ask the salesman to open the box to let you see the miraculous device. No, he says I can't do that. Why not? Well, everything you need to know is on this box. It's really a great food processor. Yes, but I'd love to, you know, plug it in, see a demonstration or something. Sorry all I can show you is this box. [MUSIC] So what does that have to do with medical studies? Well, medical studies are very much like that box that the salesman had. They are nice and flashy on the outside, and they tell you all the great things that are inside but very rarely do you actually get to see inside the box. You don't get to see the data that generated all the results in the medical study you just get to see what the authors tell you. Now the vast majority of scientific authors are honest brokers and ethical people that really do want to advance health care in the country and the world. But you do have to know that what you see is not always what you get. And part of this course is going to be teaching you how to recognize what might be lying inside that box. Now a lot of times people wonder how can we be spending so much money on research? And it's a great question, research costs a ton of money. Huge sums are invested globally in medical research and development and with good reason, right? The US spends more than a $100 billion a year on medical research and the majority of that comes from the federal government in the form of the National Institutes of Health. The NIH has an annual budget of $33 billion, so just a tremendous amount of money going towards medical research. So what do you get for that money? Well, you actually get quite a lot. And I think Mary Lasker said it best when she said if you think research is expensive try disease, right? And so just to give you a few examples of amazing returns on investment. The Human Genome Project was a $3.8 billion investment at the federal level, just an unfathomable sum of money to do a science experiment. And yet the return on that investment is about a $1 trillion worth of economic growth and the explosion of a bunch of new fields of science, so definitely worth it. On a slightly smaller scale, but perhaps no less impressive the Women's Health Initiative clinical trial which was a clinical trial which evaluated hormone replacement after menopause cost a whopping $250 million to perform. That's a huge sum of money but resulted in $37 billion of savings and economic output in the long run. So medical research costs a lot of money, but you make a lot of money too. Now how does research get from the idea phase to in your hand in a medical study? Well, there are basically two paths. One is the path of academia which is where I come from. And one is the path of industry which are where a lot of our pharmaceutical companies are lying. So, let me give you a story here. This was a paper we published back in 2016. That's me, this was one of my papers looking at a clinical trial evaluating automated alerts for a condition called acute kidney injury. And this is what we went through to get this paper published. So first we had a research question. We wondered if these alerts would benefit people. We developed a hypothesis. We said to ourselves yes, we think they will benefit people in terms of reducing the risk of death and dialysis and other bad outcomes that happen to patients who have their kidneys injured. We had an idea to test the hypothesis a randomized trial. And then you come to this critical question, right? How are we going to pay for this? As we just said this cost a lot of money, so you write a grant. Grant writing is essential to the process of academic medicine. It's what most of us spend a lot of our time doing. And the way that works is you put together a big proposal and it can be upwards of 150 pages which you submit to a funding agency. That is reviewed by a panel of your peers presumably and sent back to you, they are often rejected or called for revision. And that cycle can repeat over and over and over again until if you are very lucky and you're working on something that they are interested in your grant gets funded. Then you have the money you can conduct the study, you collect the data, analyze the data and write up a manuscript, but it doesn't stop there. That manuscript that you've written, the results of your study doesn't just get distributed via the New York Times or something you submit it to a medical journal. And medical journals all want your peers again to read your manuscript and criticize it for them. That's a process called peer review. The manuscript will go to three or four experts in the field you're talking about and the experts will right back with questions. What did you mean by this? Why didn't you do this? Don't you think this is another explanation for the results you see here? That process can iterate multiple times as well, the manuscript continues to get changed. Finally, hopefully, it's accepted by that major medical journal and the manuscript is published and that's how you got from A to Z. It will not surprise you to hear that this process can take a decade [LAUGH] between when that idea emerges in your mind and when that final paper is published. It is a slow process, but it's one that often generates good results. What about industry? So this is an article that appeared in the New England Journal of Medicine looking at a drug called Pembrolizumab to treat a certain type of lung cancer, and broadly speaking the pathways the same. Industry identifies a promising agent rather than a question really. They identify a compound that they think might be active and help treat a disease. They just go ahead and conduct the study. They've got their own money. In fact, they have quite a lot of money so they can skip over that grant writing step altogether which must be nice for them. They conduct their study, collect the data, analyze their data and write the manuscript. And then the same process of peer review applies to them until eventually the manuscript is accepted. The two major pathways leading from that first idea to that final publish manuscript that you're sitting reading when you read a medical study. Now, what is this peer review? Well, we submit a manuscript and our peers read through it. They are anonymous, we don't know who they are. And they are sometimes a bit rough when they respond to us, so here are some examples of peer review. The first one is one that I received recently [COUGH]. Why the authors think an extremely convoluted technique developed in marketing should be taken seriously in clinical research where highly competent statisticians have developed techniques over decades of research is mind-boggling. That was a less than constructive piece of criticism, but most peer review is actually pretty good. Here are some other examples that I like. Since the manuscript is so lacking in all aspects I won't bother going through it in detail, not a very productive piece of peer review. And finally, this paper is fluently written and meticulously researched. I do not recommend it for publication. So peer review hit or miss, but I show you these because they're kind of funny and it's fun to laugh about a little bit. But a lot of peer review is actually really productive and makes the paper much better. Now publication. A lot of people would assume that we researchers make money when we publish papers, but it's in fact quite the opposite. We are usually charged to publish a paper. In the bigger journals that are more profitable that charge may be waived or in certain cases those journals are free to publish in, but they are not paying you for these articles in any way. The charges can vary quite a bit depending on the journal, and we'll talk about some of those charges, especially when we start discussing this concept of open access journals. But we don't make our salary from publishing papers at least not directly. Now you might think if we've paid to have our paper published that you shouldn't have to pay to read it. Nevertheless, when you go to try to read it you may run into any number of paywalls which is of course how these journals make their money. And there's quite a bit of criticism that these journals are essentially making money off the backs of the taxpayer who is funding all of this research. And I'll tell you how as a taxpayer you may be able to get around some of these paywalls. But nevertheless, the state of play right now is that many of these journals do not give you access to this information for free at least not very easily. So let's talk about getting [LAUGH] around these paywalls, right? I'm of the firm belief that information should be free especially information that you paid for through your tax dollars, right? So if you see an interesting medical study how can you get it? How can you get it in your hands? Well, number one, you can ask your local librarian. So many libraries even just public libraries, municipal libraries across the country have subscriptions to multiple medical journals. And so you can just request this, and they will print it out for you or photocopy it for you not too hard. Number two, PubMed Central. So there is a law that says if the government funds the research the research has to be publicly available, and that is published on a place called PubMed Central. Now the catch to PubMed Central, and the link is right here, is that there's about a year, up to a year lag between when the paper's published in the journal and when it appears on PubMed Central. So if that is a hot off the press paper you want to see you may not see it in PubMed Central when you go there. Google Scholar is highly useful. If you go to scholar.google.com and type in the title of the paper you'll find multiple listings for it. And oftentimes some of those are not behind paywalls, so take a look. And number four, this is almost universally effective, email the authors. The corresponding author which is usually either the first or last author listed in the series of authors in the paper will have their email printed on the abstract of the paper, which is usually in front of the paywall. You simply email them and you say, I found your study very interesting would you mind sending me a copy? And I have never had an experience where an author declines that, so multiple steps to get around paywalls. You should not have to pay for this especially for federally funded research that you've paid for once already. Now we're talking about seeing these papers, so I want to talk to you a little bit about this concept of open access. So open access sounds great. The idea behind open access is we give you the papers for free. Open access no paywalls read whatever you want. Sounds great, right? But it's a bit of a two-faced God. So the good side is that it makes this information freely available to everyone. The papers get seen by more people, broader dissemination, more discussion. And some journals are entirely open access. A famous one is called PLOS One and you can go online you can read any paper that's there, as soon as it gets published you can look at it great. But there's a bit of a downside, right? Someone has to pay for this. So the charges to the researchers who are publishing the paper can be quite high ranging between $2,000 and $5,000 to have their paper appear open access. Now, once you introduce that kind of money they're going to be people who take advantage of that. And so there have evolved what we call predatory open access journals. These are essentially fake scientific journals that if you pay enough money as a researcher they will publish your paper. Now, why would you want to do that as a researcher? Well, it gets into some of the perverse incentives in medical research that I wanted to address before we get too deep into this whole course. Now predatory journals reach out to you all the time and ask you to submit your articles. And I'd received maybe ten emails a day from various journals like this asking to submit an article. And to show you that despite the fact that they say that your article will be rigorously peer-reviewed that is not true. This is an article that appeared in the journal International Journal of Advanced Computer Technology called Get me off Your [BLEEP] Mailing List. And the entire article just consisted of the author writing get me off your [BLEEP] mailing list over and over and over again. It was accepted after peer review and published for a fee, of course. Now this tells you something, right? The International Journal of Advanced Computer Technology sounds legitimate. It sounds legitimate to me. How do you know [LAUGH] that the journal you're reading is a good journal or a predatory open access journal? There are a number of ways to tell how good a journal is or how reputable a journal is. One way is if you know someone in the field you just ask them have you ever heard of the International Journal of Advanced Computer Technology? And if they tell you, no I've never heard of it and I'm an advanced computer technologist, then it's probably not a great journal. But you can look at other ways. There's a concept called the impact factor and you should know what that is. The impact factor is the average number of times that a paper that appears in a journal gets cited. So if I publish a paper in a journal and it gets cited over the course of the next year or so 50 times that's a high-impact paper. A lot of people are citing that paper, it's an important paper. And so if you take the average of all those citations across the entire journal you can say the average article in this journal gets cited how many times? And here's just a ranking of the top impact factors to give you a sense of where we are. So you can see the Cancer Journal for Clinicians is very high up there. That's a journal that publishes mostly review articles actually, but they do tend to get cited quite a bit. But number two is the New England Journal of Medicine many of you will have heard of that a very venerable journal. One of the top journals in medicine with an impact factor of 72. Which tells you that the average article in the New England Journal gets cited by 72 other articles over some defined period of time which is pretty high. These predatory journals might have impact factors of less than 1 which [LAUGH] is to say that the average article doesn't even get cited once subsequent to publication. So you can always Google the journals impact factor to get a sense of is this article being published in a highly reputable journal? Now I told you that this is all sort of corrupted by perverse incentives. For me as an academic, why am I paying money to publish my research? Well one is that I want to get it out there. I want to tell people what we've done. But my job is dependent on how much I publish. If I don't publish papers and obtain grants then I don't advance in my career. I don't get promoted and if you don't get promoted over a certain period of time you're kindly asked to leave. And so there's an incentive to me to publish as much as possible. There's an incentive to me to publish things that make my lab look good. That make my hypotheses look good because I will reference those publications in future grants and say, look what great ideas we have. And that's really bad [LAUGH] because we want everything to be published. If I do an experiment everyone needs to see the results even if the experiment fails or else someone else is going to do the same experiment and fail again. Now pharmaceutical companies have a different perverse incentive and it's almost entirely based on dollars. So they want to publish papers that make their drug look very good so that people will prescribe their drug. Now there are some rules in place to prevent gaming the system, which we'll talk about in a subsequent lecture. But you need to be aware of the fact that the medical science enterprise is not this pure activity from sitting lofty on high in the Ivory Tower. That there are forces at play that make some of the research you're going to read not as reliable as you may think. So our take home points for today, the medical manuscript is the endpoint of a very long process. It takes a long time to get published. You don't get to see much of that process you just get to see the box at the end. The peer reviewers don't see much of that either, right? They just are reviewing the manuscript. They don't get to see the data that went into the manuscript or very rarely do they. Systems are in place that make open data less attractive to researchers in particular the very high cost. And finally, the medical research system has many perverse incentives that encourage some bad behavior. So knowing that bad behavior is out there you need to know how to identify it, and we'll cover how to identify those behaviors through the rest of this course.
