I love science and the scientific method. I also like the line I’ve been told throughout my time studying the statistics that support these scientific studies.
“There’s lies, damned lies, and statistics”
Basically statistics don’t lie, but can be used (and have been) to tell a story that leads to potentially inaccurate conclusions. The number-one thing we learn in statistics (besides the fact that TI-89+ is my friend that will do all the hard work), is that correlation doesn’t equal causation.
And yet so often, some of the smartest people writing these scientific papers forget this!…or perhaps, because I thoroughly believe they must have some sense of sustained intelligence being able to navigate the academic and scientific community to get to a point to publish a paper, I would say that it is an intentional decision to ignore this basic rule on one of the defining pieces of their career.
Your choice. Call them unable to create logical conclusions (which they’ve been programmed to do since they showed 2 standard deviation excellence in standardized tests as children) or assume there was an intentional momentary lapse of analytical prowess to sustain their position in academia.
Here’s how most paper’s conclusions enter into mainstream:
- Paper is published with wishy-washy verbiage as to the interpretations of the study and its conclusions
- The writer of a media company reads the summary and conclusion while ignoring the rest, including reading the procedures and applying actual independent thought
- From there, the writer’s media brand (I refuse to consider them a publication on principle) will pick a click-bait headline based on whatever its algorithm recommends “for optimization”
- That story will get plastered all over the brand’s different social media outlets
- From there, it will be plagiarized about 15,000 times by copy-cat publications that add no original thought to the piece other than changing the by-line and breaking it up into a slideshow so that it simultaneously pisses off the reader and boosts its page count to pad its ads impressions
- Then, just about every person will read it in one way shape or form over the course of the next few months (these evergreen articles about studies weave in and out of different tiers of intellect at different times but are perpetually moving through consciousness)
- Finally, derivative studies, articles, blog posts, and comments will cement and immortalize the logically questionable conclusions with no one really reading the original study or questioning the procedures…except of course those who are professionally trained to do it (aka the few surviving journalists hidden in remote regions somewhere), and those who oppose the conclusions because it opposes their view of the world aka their vested interest (who then counter with an equally questionable counter study)
I sound like a cranky old-grandpa on a rocking chair bemoaning the good ole days of scientific and journalistic integrity…yes, yes I am in fact drinking my prune juice and heading to bed after Jeopardy….
Here’s some fun things to fill your head with though and let you make your own conclusion. (I know it’s rare to have to think about a conclusion yourself and not have it stuffed in your face)
Academic studies won’t go very far if the hypothesis is not proven and/or there is inconclusive evidence for the hypothesis. Aka if the study doesn’t prove anything, the researchers are toast! Think about it…
The approach of “give me a PhD or tenure for my work studying X which I do not really have conclusive evidence on nor a better understanding of” is not the best way to gain your Doctorate or build job stability. Nor is it an effective way to create esteem for your university that is requiring you to publish a certain number of articles and studies each year to help boost your college’s random ranking. A ranking which, because of the lack of actual analytical skills taught in school, somehow makes sense for high schoolers to base their decision on where to apply to and pursue a lifelong journey of indentured servitude through debt.
…
So the scholars have to find some way of making their research relevant by offering conclusions which might indicate potentially revolutionary insights and suggested reasons for further studies (which of course is how to ensure job stability and be considered an “expert” in a field in which no one really will question their hyper-niched, subject-matter expertise).
So only “revolutionary insights” where there is “some reason to believe something” is going to work for the researcher’s and their university.
Unfortunately though, the conclusions with some caveats in them like “some reason to believe” when read hastily by the media brands and social blogosphere of immature amateur writers (myself included), transforms itself. With all due liberty (#doubleEntendre), media brands now change “some reason to believe” into “omg guess what everyone, click here because science just blew my mind and it will blow yours too” And of course, you will click on it, because well, the headline was made by science (#doubleEntendre)
I mean you can’t blame us amateurs…it’s not like we have an editor to report to who is trying to uphold decades of the publication’s journalistic integrity….
Let’s tack an MBA approach to this: Just follow the cash flow and incentive structures.
To a man with a hammer, everything in the world looks like a nail.
Don’t blame the academics who by nature are risk averse and just trying to make sure they continue to be employed.
Don’t blame the bloggers for being sensational, for sensationalism is what we deep down are all attracted to and the way these bloggers got their rise to power…toppling the ancient dinosaurs wielding their stagnate moral compass and archaic due diligence process.
Hmm, that’s a fun story you’ve weaved Clifton, but I’m not sure there’s ample data to suggest this…
Fair point, loose data points strung together in a narrative does sound slightly suspicious and should warrant further investigation. How dare I!
..oh wait, that’s exactly how agendas get formed through the American narrative and pop culture!
That’s a heavy one. Let me repeat it: the common “hot topics” permeating through America’s collective psyche are formed through stringing together loose data points derived from scientific study. The newest fad, recommendation by the Surgeon General, and anything mentioned on Dr. Oz and the Tonight Show are typically solid examples of these “hot topics”. They are in fact alluding to scientific studies and pick out several of them to prove their point, so what’s wrong with that? Well…
The beauty of the scientific method is that:
- It leads to the revelation of truth, which means if an experiment proves counter to the narrative it should reveal the inaccuracy or modification needed to that narrative.
- A factual conclusion should be able to be replicated with a similar outcome.
The challenge with point 1) is that there are thousands of studies being performed every year and published. Only a small fraction get selected to enter into the mainstream’s narrative. Selective data points (here the selectivity being the actual studies being identified and discussed) in itself is a violation of the scientific method.
Picking only the studies with the conclusions you want to talk about while ignoring the rest is the definition of bad science.
In that same approach, based on anecdotal evidence (read can’t be proven so ignore this if you want, but probably wise not to)…I have had numerous discussions with PhD researchers who’ve had their studies funded by specific interest groups…and let’s just say these interest groups don’t exactly fall into the category of “solar-power generating, homeopathic using, organic vegetarians, who walk to work”.
Here’s what happened: When/if a conclusion these researchers had were demonstrating a conclusion contrary to the conclusion desired, there were incentives both positive and negative to stop, disappear or amend the study.
“Prove this is safe and effective” is probably not the most impartial way of starting to conduct an “unbiased, scientifically sound experiment”
If the outcome was contra the funders’ desires, poof goes the study into the ether and/or woosh goes the signing bonus way high to keep the study in the academia equivalent of purgatory.
So what’s left? Cherry picked studies to string a narrative that’s beneficial to the funder of the studies. Fat Tony logic running the streets would be, “Ya duh, why would we invest in a study that would hurt us? What do you take us for stupids?”
Going back to pseudo-journalistic logic, do I have a data-set large enough to justify an extrapolation of these discussions I’ve had with researchers to apply it to the whole of the scientific community or even a specific industry?
Wait, let’s talk about that shall we: sample size of data-sets…I promise to make it fun 🙂
Meta-studies (those studying the big picture of studies on a particular area) have consistently shown that a lot of these “cherry-picked studies” in our national narrative have trouble being duplicated.
Wait what? The studies so often referred to when repeated are hard to get similar outcomes.
Dropping a book and a ball so that they land on the floor at similar times is something that is repeated in elementary school science classes all over the world every year. These studies if their right should be replicated, so why is it so hard then to get the similar conclusions on these fancy studies?
Well let’s go back to what I mentioned before regarding the lie of statistics. There is something called p-hacking (Google it).
One could get the desired conclusion by playing with the data, including the number of samples needed for the statistics to work. This means then that when the study is trying to be recreated, perhaps there would be different conclusions or more likely, inconclusive evidence to support the conclusion made. And that’s what a lot of these meta-studies are proving!
Oh man, you’re totally buzz killing all my hard work to make my study look legit and get me my PhD/tenure…why you gotta be like that homie? that’s so not the bro-phd-code…
And let’s be clear here, this argument applies to both sides of an agenda, regardless of the side you’re on it’s being done.
So are you arguing that all of these studies should be discounted and that they in themselves provide so much uncertainty that they should be discredited entirely?
Yes.
…seriously?
Okay maybe credited a bit…kinda like the credit you give your friend at a bar who spouts off a random fact of a story you were a part of and you know he or she is not exactly telling the story how it went down, but it does make for a better story if there were 5 dudes you stood up to instead of 3…
Let’s recap. There’s been significant evidence to show that there are numerous ways to fudge the numbers, to ignore both sides of the truth, to exaggerate the findings. There lacks integrity and consistency in the micro and macro trends of scientific studies. That’s a tough fudge to swallow.
Instead of being the ultimate journalist/academic who points out only flaws and offers no solutions…I’m, in fact, going to recommend some:
The first, read Nassim Taleb’s Skin in the Game. He backs up his statements and takes far more punches at this than I did, such a good read! (And yes it is available on Audiobook for those who want to cheat, but it only counts if you have good listening skills…which means it’s inapplicable to a majority of the heterosexual male community, so buy the book! #stereotyping).
Second, trust yourself. Trust your experiences. Trust your intuition. What Taleb calls “Fat Tony” logic, is a great check to make sure that we haven’t gone too far down the rabbit hole of the minutiae and academic theory that when we put it all together and try to apply it in the real world it doesn’t pass the smell test.
I don’t care how many studies were made on the expiration date of milk saying milk lasts this long, this specific bottle of milk smells rank and has gone bad, if you trust the expiry date so much, you drink it, not me.
Conclusions: For those who enjoy jumping to conclusions (#irony), my main point is to imbue you with a sense of healthy skepticism for scientific studies. They have been hijacked by p-hackers, faulty logic, special interest funders, bloggers and social media. They are following the same demise of trust as journalism. It’s the same fact pattern.
When you read a headline about a recent study, go to the actual study and read the procedure section, don’t just rely on headlines and conclusion. Think for yourself whether the procedures are valid or not. See if there has been an attempt to repeat the findings in another study, perhaps funded by the opposing agenda. And finally, check in with what your gut says, does it pass the smell test?
Which brings me to this: if you don’t have a healthy gut (which a majority of American’s don’t due to overuse of antibiotics, prescription drugs, and chemical intrusion into our food supply…and thinking Kombucha will solve all your problems–it doesn’t), work to repair your gut so that it can actually talk to you and help guide your decisions properly (I went biological to metaphysical there…)
At Human Garage, we’ve been able to help reverse autoimmune issues, restore gut health, and help make sure you don’t fall for the constant noise of the faulty American Narratives. #plugover #thinkForYourself