Why Omega-3 Study Design Matters.

OmegaMatters: Episode 6 

Hosts: Drs. Bill Harris & Kristina Harris Jackson

Guest: Nathan Tintle, PhD

Background and key takeaways:

The way studies are designed, especially when it comes to omega-3 fatty acids, is very important when it comes to pinpointing their value in human health. However, because they are nutrients and not drugs, they don’t always hold up when it comes to the gold standard (i.e., randomized controlled trial, or RCT) way of studying substances and measuring them against health outcomes. Sometimes different kinds of studies can yield better learnings about nutrients, such as prospective studies (looking in the future) and retrospective studies (looking back in time), and then there is always the observational study that has a power all its own. In this episode, Drs. Bill Harris and Kristina Harris Jackson go deep into the world of research to understand the nuances of clinical studies with the help of biostatistical expert, Dr. Nathan Tintle. He says the two most important questions you need to ask when reading a study are: 1) Who are the people in the study and can I generalize the findings from them? And 2) Can I make a cause and effect conclusion from this study? One of the advantages of these other studies (i.e., prospective, observational, etc.) vs. an RCT in nutrition, and particularly in the omega-3 space, is that there are better conclusions to draw about prevention. In other words, the majority of omega-3 studies seem to point to benefits for those who have maintained a lifetime of high omega-3 levels.

Guest Bios:

Dr. Tintle holds a PhD in statistics from Stony Brook University. He has internationally recognized expertise in fatty acids, public health, genetics, and biomedical statistics via study design, survey instrument development, and standard and computationally intensive data analysis techniques. He has >100 peer-reviewed publications. Dr. Tintle also brings prior relevant experience as an executive director of a non-profit, research institute. As a researcher and research institute director he has led initiatives securing over $8 million of funding across 30+ major federally funded grant awards, in addition to securing substantial private and public charitable donations.

Transcript:

Dr. Kristina Harris Jackson: Welcome to Omega Matters where we have casual conversations about science and fatty acids and other interesting things. I’m Kristina Jackson, I’m the Director of Research at OmegaQuant Analytics, which is a clinical lab that specializes in fatty acids, and I’m also a registered dietitian. We also have Dr. Bill Harris who is now the president and founder of FARI, the Fatty Acid Research Institute, a non-profit conducting studies in on fatty acids, particularly omega-3s.

And we are joined today by Dr. Nathan Tintle. Nathan is a PhD in statistics, so we have a great conversation today about statistics and how to read scientific literature. Nathan is internationally recognized in fatty acids, public health, genetics, and biomedical statistics, study designs, survey, instrument development, standard and computationally intensive data analysis techniques. He’s been on hundreds of peer reviewed publications and he also is the Executive Director of FARI.

So, we are here. All of us are actually up here in the Dakotas, technically. Are you technically Iowa, Nathan.

Dr. Nathan Tintle:  I’m across the border in Iowa, but not too, not too far away from South Dakota.

Dr. Kristina Harris Jackson:  Yeah, and we’re getting our spring storm the rest of this day.

Dr. Nathan Tintle:  Yes.

Dr. Kristina Harris Jackson:  So it’s a good day to be on Zoom.

Dr. Nathan Tintle: Yes.

DISCUSSION:

Dr. Kristina Harris Jackson: Today we really wanted to have Nathan on early on in this series to start a conversation about statistics and study design because that has everything to do with how we design studies and read studies and know what we can and can’t say about a study. After we do a little bit of foundational work, we’re going discuss a specific paper to kind of show, how, how these principles affect how we read papers in real life. So, with that, we can get started.

There has been a lot of media reporting on scientific papers this year. With the pandemic, everyone’s just gotten a lot more used to reading more scientific things. Knowing a little bit more about science and study design can help you understand studies that you hear about in the media. So, our first question today is what does study design have to do with how good the science is? And we’re going to focus mostly on health, fatty acids, that area, but we’ll bring in examples from other areas. So, go ahead, Nathan.

Dr. Nathan Tintle: Yeah, well thanks again for the opportunity to talk today, and just for the audience out there, you know, most people when they hear statistics, it’s not really usually a real positive reaction. And so, but you’re right that the last year has really just thrust to the forefront of how data can help us make decisions, right? And, especially in the midst of crises. So I think it certainly has raised the awareness for how important good quality data is and our ability to interpret that data and decide what’s good quality and what’s not good quality. As you said, it’s important for all of us just as citizens in the world, but especially when we look at public health in fatty acids and things like this pandemic to be able to vet that ourselves, so we can make decisions on an individual basis, about our health and our behaviors.

When you’re starting to look at a study, there’s two questions you should ask about every study you hear. The first question has to do with what I would call generalizability. So, who are the people in this study and how much are they like other people we want to learn about? This question of representativeness is often really important. Maybe the study was done on older folks and you’re curious about how vaccines or how a drug treatment works in younger individuals or middle-aged individuals, and in that case, it’s not necessarily going to be the same. And so, you need to take that into account. The best studies do things to try to make them generalizable. And the best way to do that is by using random sampling.

For example, recently the Omega-3 index was measured on a random sample of Canadians — giving us a country-wide sense of omega-3 levels. Up until now, it’s been really easy to get suggestions of that, but often those suggestions are from very biased samples.

Dr. Nathan Tintle: People who are volunteering to have their Omega-3 index tested can give us senses of maybe what those Omega-3 levels are, but until you really do a systematic random sample of some kind, you don’t know for sure.  A random sample is going to allow you to generalize. Otherwise, there’s a potential for bias. So we’re back to: Who are the people in the study and can I generalize the findings from them?

The really important question to ask is can I make a cause and effect conclusion from this study. So, if we say, for example, that this omega-3 supplement works, then this is the effect you will see. In other words you need to ask yourself: “Well, how do we know that that’s a cause and effect relationship?” And the best way to do that is using something called random assignment.

You’ve probably heard the term randomized clinical trial. The idea of randomization is different than random sampling. It’s random assignment, which means the people in your study will be randomly assigned to get a treatment or a vaccine or a drug, while the other group gets a placebo. And then, after a while, we’re going compare those two groups.

So, why is random assignment so important? It’s so important because it keeps those two groups the same initially. So, your placebo group and your treatment group are going to look the same because you’ve randomly assigned people to them. And that’s important so you don’t have this thing called confounding variation, or confounding variables, that often come into play.

So, there’s been a lot of research and ongoing debate about alcohol consumption and health. And so, maybe you see a study that says, “Hey, people who drink two or more glasses of wine a day tend to live longer.” So the question we should ask is: Did that study allow us to say that alcohol consumption causes you to live longer. You say, “Well, probably.”

However, this wasn’t a study where they randomly assigned some people to drink two glasses of wine a day for 30 years of longer and other people not to. It would be very hard to do that. And it wouldn’t be very cost effective. So this probably wasn’t random assignment, which means that other things could be better explaining that relationship.

For example, what if people who have two glasses of wine a day have better access to healthcare or they also eat healthier in general or they exercise more? Any of those might be explanations as to why those people are living longer. And so, this question of was there random assignment gets at is this potentially a cause and effect relationship. And that’s, of great importance, just like our ability to generalize is.

Dr. Bill Harris: I like to call those studies “people who” studies because they’re studies about people who do something or people who have something. You know, people who drink alcohol, people who have a high Omega-3 level, or people who smoke. The problem with “people who” studies is that you can’t determine cause and effect in those studies.

Dr. Nathan Tintle: Right, yeah.

Dr. Kristina Harris Jackson: The gold standard for studies is the randomized controlled trial or RCT. But as you’ve kind of just alluded to, it’s not possible in the real world to do RCTs for every question we have and especially in the world of nutrition or long-term health outcomes.

Dr. Nathan Tintle: Yeah.

Dr. Kristina Harris Jackson: Can you talk a little bit more about that side of the equation in the health field?

Dr. Nathan Tintle: Yeah, sure. Like you said, I mean, randomized controlled trials really are, in a lot of ways, the gold standard. But you’re absolutely right that there’s times where it’s just not possible. And so, you know, one of the questions I will often ask is: How do we know that smoking causes lung cancer? I mean, most people would say, “We know that smoking causes lung cancer.” It wasn’t from a randomized controlled trial where they had some people smoke and some people didn’t smoke and they tracked them for 30 years. It was from a lot of studies, many studies, some of them were pretty complex, but studies over time — prospective trials. We follow people you know, a group that’s maybe healthy, maybe just a representative sample, we follow them for a very long time and then we see what happens. And so, randomized clinical trials, yes, they’re good, but they’re not perfect.

There is another limitation of this type of clinical trial, which is, oftentimes, a randomized trial does not have a representative sample. They may, in fact, end up being a very biased group of individuals. People who volunteer, for example, to participate in a nutrition study or a smoking study, often have a lot of motivation to participate in such a study.

So if we say, “Oh look, this new smoking cessation program to help people not get lung cancer worked really well in these folks, better than say, a placebo. You say, well, those were people who said yeah, I’d love to participate in the smoking cessation study. So, maybe they had a lot of internal motivation and that might change your ability to say, who are these people and will this really work as effectively in a general population as it might in this particular group.

On the flip side, as you’ve already alluded to, Kristina, in some of these nutrition settings we look at long term health outcomes like mortality, like cardiovascular disease, like cancer, and in these cases the prospective trial is really the way to go. Where you look at a group of individuals and you follow them for decades and decades. And all that time, you’re measuring a lot of things and that’s really critical, so that when you see different outcomes occur in those folks, you can start attributing that to different potential factors. This is where there is the potential there for these confounding variables to creep in.

We were talking about that in the case of the alcohol example, but there’s things statistically that you can do. They’re not perfect, they’re not as good as a randomized trial, but they can end up pretty good, in these prospective trials at eliminating other potential explanations and zero in on what you think is pretty close to the cause and effect conclusion.

Dr. Kristina Harris Jackson: I like to think of the different study designs as an ecosystem because they all have their place We start somewhere with our questions and a lot of times, it starts with observational studies, which are the bottom of the totem pole, and those studies can lead you astray, but they also can start you down the path of getting more and more sophisticated studies and finding something real. So all of these different study designs have a role. It’s just we need to be aware of what the pros and cons of each one are.

Dr. Bill Harris: Definitely.

Dr. Kristina Harris Jackson: Communicating science is hard and you have to get to the point of what the study says.

Dr. Nathan Tintle: Yeah. I’m just going to share my screen here for a minute and show a little chart that we use to help us, at least initially make sense of some of that. And you’re right that, you know, it can get really complicated and overwhelming. When I talk to research students and laypeople, they are often they’re very overwhelmed, that they can’t even begin to understand this. So one of the things we talk about is how can you ask some fairly straight forward questions initially to start understanding it? Because we also want to make sure that we’re not just saying, well, you know, like in a pandemic, I don’t know what the data’s telling me, so I’m just going to do whatever I think is right or what you see at face value.

Dr. Nathan Tintle: So this little grid here kind of goes back to those two questions I was talking about, and I really do think that that’s the best way to start making sense of things. And, yes, there’s a whole lot more complexity behind it, but this question of, you know, how did we get these groups we’re comparing? Did they randomly assign people to drink two glasses of wine or not? Did we randomly assign people to smoke or not? Or did we randomly assign some people to get a new vaccine and some people to placebo group? And based on that, if you’re randomly assigned, then cause and effect conclusions are possible. They’re not guaranteed, but they’re possible.

But if you didn’t have random assignment, then there is this potential for confounding. As we talked about with the random sampling as well, how did you get the people into the study? Who can we generalize to or not? And as you were alluding to a minute ago, Kristina, often we’re starting down here in kind of this bottom right grid or bottom right section of the grid, where we can’t generalize very well and we can’t draw a cause and effect conclusion. A lot of the studies out there aren’t randomized trials and they’re using a convenient sample of people.

Now, before we think, well, why do we even bother doing those, they’re the most pragmatic. They’re the cheapest to do, and I would make the argument that’s often where really important scientific questions get asked. Because it’s those really risky studies, the ones you say, wow, I wonder if that thing is true, where you can’t maybe justify a really big, costly, randomized trial, and you can’t really justify a really systematic sample. But you might be able to do something pretty inexpensively that gives suggestive evidence.

And then you kind of work your way up the grid, up towards the top left, where it’s really the optimal scenario of randomization and random sample. But the reality is those hardly ever happen in practice. They’re really practically challenging to get a random sample and do the random assignment.

And so there’s also times, whereas we talked about with smoking, where it’s not even really possible to do it in a practical way.

Dr. Kristina Harris Jackson: Yeah. You don’t want to force people to start smoking for your study.

Dr. Nathan Tintle: That would generally be a bad idea.

Dr. Kristina Harris Jackson: Ethically, it’s not going to work, so yeah, those are great questions, and a great way to- to start thinking about studies.

Should we start looking at the paper for today and apply some of these questions? So the paper we’re going to look at is a paper that Dr. Harris and Dr. Tintle published in 2018, in the Journal of Clinical Lipidology. It’s titled Erythrocyte Long Chain Omega-3 Fatty Acid Levels are Inversely Associated with Mortality and with Incident Cardiovascular Disease. This was from the Framingham heart study.

So who would like to open up, with what were you trying to answer with this question in this paper. Also discuss a little bit about why the Framingham heart study is a unique group of people.

Dr. Bill Harris: Yeah, I can handle some of that. This is a study that was set up through a grant application that we filed with the National Institutes of Health and funded about 10 years ago to take blood samples that had been collected in Framingham, a suburb of Boston. It’s a city where in 1948 the first major prospective population study of what causes heart disease was done. And that was the goal of the Framingham heart study, where they recruited around 4000, men and women in that town, a pretty much a random sample, like Nathan was talking about.

They took blood samples from them and all kinds of medical histories, and then they just watched what happened to them over the next several decades. The study continues to this day, except most everybody died in the first group, so now they have a group called The Framingham Offspring study, which started in the 19070s and is made up mostly of children from the original study.

We have blood samples from one of the exams of the offspring cohort, so roughly around 3000 or so samples from men and women who were in their mid-60s on average. We analyzed their omega-3 fatty acid level. We know their omega-6 and the other fatty acids as well. So roughly 15 years ago, we measured their omega-3 status and then asked the question: Who is still alive in this group of people and does that relate to what their omega-3 level was? And our hypothesis was that people who had the highest omega-3 levels back in the 2005-2006 era would be more likely to be alive today than people who had low omega-3.

Dr. Nathan Tintle: Sometimes you have established studies like the Framingham heart study and you have the right data available. You have those baseline measures of omega-3 and you have a study that’s been ongoing, as Bill said, since the 1940s, and they’re still tracking people. And they’re really doing a very careful job of monitoring people. And so, you know, looking at, and knowing very precisely, if they died, when they died, and what other medicals issues they have — cardiovascular disease, cancer, and all these different things — you have a really rich data set.

To some extent, then you say, well, if we want to ask this question, what data sets are out there? And it’s really unique to have a data set of this quality with all these different things measured on it. And in knowing that because, as we talked about before, this is a prospective study, we’re following people forward in time, and that’s the appropriate kind of study for an outcome like mortality, where you’re following people forward in time and asking those questions.

Framingham is a highly regarded cohort. The sample size is pretty large. I mean, 3000 people is- is pretty good for studies of this nature. And there’s lots of good quality data. Not only the omega-3 levels and whether or not and when people died, but also all of these other potential things that could be confounding variables, right? So, things like alcohol consumption, BMI, smoking, etc.

That allows us then statistically to account for those things and say, well, we see this relationship, and it’s not better explained by BMI. It’s not better explained by alcohol consumption. It’s not better explained by income level. And we can go on and rule out of a lot of those potential explanations, which gets us closer to saying this is, you know, close to a cause and effect kind of conclusion. Not there yet. It’s not a clinical trial. But we can start bumping up against that kind of conclusion.

Dr. Kristina Harris Jackson: Yeah. I really like the visual of you have a randomized controlled trial, and you’re trying to match the groups through that process. You try and match them so they’re really similar, and you try and do that process through statistics when you can’t do that. And so you control it either in study design or you control it with statistics, and that gives a lot more confidence to the main variable that you’re looking at.

And the other note I wanted to make was, in case you didn’t know, the Omega-3 Index is something that we study at OmegaQuant, and having that biomarker as the biomarker of interest is also really important because it’s appropriate in establishing long-term status. So you want to make sure that if you’re looking at long-term outcomes that the biomarker that’s relatively stable and isn’t going to be jumping up and down all day and is not collected appropriately. Having that original biomarker is really important when you want to apply it to these very long studies with hard outcomes, like death. It’s a big deal. So this was a really great study for us to be able to analyze omega-3s in. So what are some of the main conclusions out of this study that you guys would like to highlight?

Dr. Bill Harris: So, let’s see. I guess, here is the abstract of the study. So basically we said, okay, we’re going to take everybody who we have blood samples for in the Framingham Study that were drawn right there 2005-6. Then we’re going to ask who of them are still alive at the end of when we did the analysis. So this was called a prospective study, meaning we went forward in time. You can do retrospective studies, and, Nathan, what’s a retrospective study and what’s the disadvantage of that?

Dr. Nathan Tintle: A retrospective study, you know, once you see cancer as an outcome, you might look at people who have cancer now and then try to find people who look like those with cancer except without cancer. So, you know, similar age profile and other characteristics. And then try to look back at, despite some of the similarities now, what things are different about them. One of the challenges when you do that kind of comparison is you often end up with groups that aren’t as representative as you’d like, and so as Kristina alluded to a minute ago, and you alluded to earlier, Bill, right, this group of individuals was, at least at the time, you know, fairly representative of Framingham, so you’re less biased in terms of the sampling perspective. So for example, in this case, you have omega-3 levels measured here, well before any of the people died. In a retrospective study, you might not have that advantage, especially if you decide, hey, let’s go look at some people who have this cancer now and compare them with some people who don’t and see how different they are now, that may tell you a very different story, so if there’s different metabolic changes going on because people have cancer, they’ve changed their behaviors. You might not be able to say, “Well, what were those risk factors 15 years ago?”

Dr. Bill Harris: Yeah, yeah. Good point. The other thing about the design of the study is we call it observational, so as opposed to what?

Dr. Nathan Tintle: In an observational study we’re not intervening, so, you know, we’re looking at the omega-3 levels here but it wasn’t like anyone went in and said, “Okay, you take more omega-3 supplements,” or “You eat more fatty fish.” It was we’re just going to see what that is. We’re going to kind of just sit back and observe what happens in these folks over the next 15 years, as opposed intervening, and then if we do intervene, doing something with random assignment.

Dr. Bill Harris: So then just to follow the abstract, our exposure marker (i.e., Omega-3 Index), we called it exposure because that’s what the person was exposed to, right? They weren’t exposed to cigarette smoke. They were exposed to a certain level of omega-3 in their blood. And then we measured that at baseline. And then the outcomes are, again, mortality—death from cardiovascular disease, death from anything, death from cancer, death from everything else. We also looked at total cardiovascular disease events.

Dr. Nathan Tintle: In terms of cardiovascular events, we’re talking about things like acute myocardial infarction or a stroke, and so then we grouped those together and said that if people had either of those, either a stroke or a heart attack essentially, then we’d call that a totally CVD.

Dr. Bill Harris: Right.

Dr. Bill Harris: So an, an event doesn’t have to end in death?

Dr. Nathan Tintle: Correct.

Dr. Bill Harris: We followed up for 7.3 years. And I guess when we started this analysis that’s about as much follow-up data as we had available.

Dr. Nathan Tintle: Right. It takes them a couple years to really do what we would call a careful adjudication, especially on the cardiovascular disease side. So, as, as you pointed out, right, we predicted both total mortality, as well as cardiovascular disease risk. And, you know, the idea of getting that data on people, getting it sort of reviewed by medical experts, combing through, you know, death records and medical records, does take a fair bit of time on a study of, of this size. And so, there’s a bit of a time lag yeah.

Dr. Nathan Tintle: I was alluding to this a little while ago that when you have a prospective study like this, and you’ve measured a lot of things on it, it allows us statistically to say if we see a relationship, and we did here between the omega-3 and CVD mortality. It is better explained by something else like BMI. Well, we can statistically do something called a multi-variable model, also called adjusting for something like BMI, and essentially say, well, even if BMI is associated with risk of death or cardiovascular disease, the relationship we see between the Omega-3 Index and CVD or mortality is not better explained by BMI. So it’s not just something like people with lower BMI have higher omega-3 indexes, and that’s why you’re seeing a relationship with the Omega-3 Index. In an observational study like this, you can’t rule out all those confounding variables by design, because you don’t have the random assignment, but you can do it statistically.

And so in this case we looked at 18 other potential explanations for why there would be a difference in risk of death or risk of cardiovascular disease. And as Bill will talk about in a minute in the results, you know, sort of after we did that adjustment we were left with being able to say independent of all those things, what kind of predictability or what kind of relationship is there between the Omega-3 Index and our outcome of interest.

Dr. Bill Harris: So let’s get down to our findings here. This table here shows all the factors that we were adjusting for, education, employment, health insurance status. We have diabetes, taking cholesterol meds, are you a smoker, blood pressure, things like that. All things that could play a role.

Dr. Nathan Tintle: Yeah, and the other thing that does, Bill, is that also gives us a sense of who these people are, right? And so they were representative of Framingham, Massachusetts, but, you know, it tells you things about their income status and how many people had diabetes in the study and things like this, which do impact our ability to generalize.

Dr. Bill Harris: So here is the figure that looked at. Nathan, I’ll let you explain it.

Dr. Nathan Tintle: So what we’re trying to look at here is how, as your Omega-3 Index changes, and we’ll just focus on the any mortality, group of little bars over on the right side, how does your risk of death change based on your Omega-3 Index values? And so what we did is we started by grouping the Omega-3 Index into five groups of fairly equal size. We call them quintiles. So we see that about 20% of the sample had an Omega-3 Index level less than 4.2%, and then we go to 4.2% to 4.9%, and all the way up to the highest group, which has an Omega-3 Index level of 6.8%. And then we ask the question: How different was risk of death, so mortality, across those different groups? And when you do these kinds of comparisons, you usually pick sort of a baseline group. How different was, for example, your 6.8% or higher Omega-3 Index group relative to that lowest Omega-3 Index group. That is that bar all the way to the right of the screen, in light blue.

Dr. Nathan Tintle: So that one all the way to the right is around 7ish, so that group is 7 times as likely as the lowest group, that darker blue bar, which is at 1, our reference group. So essentially the fact that all of those bars, aside from the dark blue one, are less than 1, they’re all below the one in dark blue, that tells us that as that Omega-3 Index goes up, risk of death decreases.

Dr. Bill Harris: People who have a higher Omega-3 Index versus people who have a lower one, there’s a difference in risk of death.

Dr. Nathan Tintle: Right.

Dr. Kristina Harris Jackson: Because of the study design, we’re able to say actual risk of death was lower in people who had higher omega-3s earlier in their life versus people who had lower omega-3s. So there’s assumptions built into that as well, you actually have a hard outcome over time and it’s been adjusted for a lot of the things we know also affect death. So it’s a strong study design. It bumps you closer to an RCT type of study, but obviously you can’t do RCTs for decades. We do a lot of these kinds of studies with the Omega-3 Index. And there’s more and more of these really big datasets getting pulled together to ask more of these kinds of questions to get more data that’s generalizable.

Dr. Bill Harris: I think one of the advantages of this compared to a randomized trial in nutrition, really particularly in the omega-3 space, is we assume, and I think with good evidence, that the Omega-3 Index reflects years and years of a certain dietary pattern, particularly certain levels of omega-3 in the blood that cannot be replicated by taking people in their mid-60s and all of a sudden putting them on fish oil.

Dr. Kristina Harris Jackson: Yeah.

Dr. Bill Harris: That whole scenario of years and years of high omega-3 — we’re really talking about prevention.

Dr. Bill Harris: In the long run that can’t be replicated in a big, randomized trial, unless you start with people who are 25 and literally randomized them to omega-3 or not, and then follow them for 40 years. Nobody is going to do that. So this is really the best we can do and it’s held up in multiple settings.

Dr. Kristina Harris Jackson: We mostly like to look at omega-3 studies. There’s so many to look at, but in OmegaMatters, knowing the different study designs is going to matter a lot in how we talk about the studies and what the outcomes actually mean.

So that’s all the time we have for today. Thank you, Nathan, Professor Tintle, for your statistical knowledge and teachings. I’m sure we will have you back to help get even deeper into the data on some of these omega-3 studies.

Dr. Nathan Tintle: Happy to do it, yeah.

Dr. Kristina Harris Jackson: That’s all we have for today, and we’ll see you next time.

Dr. Nathan Tintle: Thank you.

Dr. Bill Harris: Bye-bye.