How to Credibility-Check Health News Like a Data Scientist?

A first-of-its-kind 2-step litmus test for answering the crucial question: "Will that study actually work for me?"

---

Prefer to watch instead of read? You can find a conversational, 10-minute video walkthrough of this entire post here

What is New?

Most study results apply to far fewer people, including the study’s participants, than medical science admits. The phenomenon of non-ergodicity is its unacknowledged blind spot.

This post introduces a simple 2-step Plausibility & Translatability Check. It allows you to vet a study’s claims like a data scientist and to uncover what works for you.

Why It Matters

Blindly following advice based on “average” results wastes your time and frustrates you when interventions don’t work. This framework helps you make smarter bets on what will actually improve your personal health.

Your Takeaways

An eye-opening understanding of the natural phenomenon of non-ergodicity and how it shakes the foundations of biomedical research. Plus a downloadable Excel tool to kick the tires of any medical/health study:

• Plausibility Check: Are a study’s foundational data believable?

• Translatability Check: Will you be able to reproduce the intervention’s result?

Table of Contents:

• The Study That Was Too Good to Be True

• Why “Normal” is the First Thing to Check

• Becoming a Data Detective: Uncovering the Hidden Shape

• The SPRITE Test: A Tool for Unmasking The Truth

• Your Plausibility Toolkit

• How to Find the Numbers in a Study

• The Translatability Check: Will It Work for Me?

• From “Will It Work?” to “What Are My Chances?”

• Your Translatability Toolkit

• Conclusion: The Only Gospel is Your Own Data

• Cited References

The Study That Was Too Good to Be True

In March 2024, the British Medical Journal (BMJ) published the results of a weight loss study that should have made the CEOs of Eli Lilly and Novo Nordisk fall into deep depression.

Screenshot taken by the author

The researchers discovered a cheap natural supplement that helps overweight youngsters (the oldest aged 25) shed 9% of their body weight within 3 months, and without any changes to their diet or physical activity [1]. That’s 50% more than the average 6% weight loss achievable with Ozempic & Co. within the same time frame [2].

So, here we have on one side the decades-old search for a slimming drug, a search that has consumed billions of dollars, millions of lab hours, and thousands of researchers. On the other side, we have apple cider vinegar (ACV), three researchers in a Lebanese lab, and a ‘discovery’ that gives the pharma giants a run for their money.

Believable? With BMJ credentials and peer-review laurels, we should take it seriously, right?

Spoiler alert: the BMJ just retracted that paper.