Normal Curves: Sexy Science, Serious Statistics

How do you decide whether a clinical trial “worked”? In Part 2 of our Galleri series, we examine the landmark randomized trial of a blood test designed to detect more than 50 cancers. We explore why different outcome measures led to dramatically different headlines, discuss primary versus secondary outcomes, pre-registration, hierarchical testing, and post hoc analyses, and explain why mortality remains the outcome everyone is waiting for. Along the way, we uncover a statistical mystery involving dozens of missing cancers and discover how a little arithmetic can sometimes reveal more than a press release.Statistical topicscancer screeningexploratory analyseshierarchical testingmissing datamultiple testingoutcome measurespost hoc analysespre-registrationprimary and secondary outcomesrandomized clinical trialsscreening testsMethodologic Morals“When the simple numbers don't add up, pay attention. The arithmetic may be trying to tell you something.”“The first question should not be, did it work? It should be, what counts as success?”ReferencesGiridhar KV, et al. Safety and performance results from PATHFINDER 2, a registrational study of a multi-cancer early detection test in an intended-use population. Presented at the 2026 American Society of Clinical Oncology (ASCO) Annual Meeting. May 2026.Hubbell E, Clarke CA, Aravanis AM, Berg CD. Modeled Reductions in Late-stage Cancer with a Multi-Cancer Early Detection Test. Cancer Epidemiol Biomarkers Prev. 2021;30(3):460-468. doi:10.1158/1055-9965.EPI-20-1134Neal RD, Johnson P, Clarke CA, et al. Cell-Free DNA-Based Multi-Cancer Early Detection Test in an Asymptomatic Screening Population (NHS-Galleri): Design of a Pragmatic, Prospective Randomised Controlled Trial. Cancers (Basel). 2022;14(19):4818. Published 2022 Oct 1. doi:10.3390/cancers14194818ASCO slides: https://grail.com/wp-content/uploads/2026/05/Swanton_ASCO-2026_NHS-Galleri_FINAL-Slides-05.26.2026.pdfUK registry protocol: https://www.isrctn.com/ISRCTN91431511 Clinicaltrials.gov protocol: https://clinicaltrials.gov/study/NCT05611632 Common biases in cancer screening studiesCancer screening studies are subject to several well-known biases that can make a screening test appear more effective than it actually is. Three of the most important are:Lead-time bias: Screening advances the time of diagnosis, making survival from diagnosis appear longer even if the patient's lifespan is unchanged. For example, if a screening test detects a Stage II cancer at age 60 that otherwise would have been diagnosed because of symptoms at age 62, but the patient dies at age 68 regardless, survival from diagnosis appears to increase from 6 years to 8 years even though the patient did not live any longer. Length bias: Screening preferentially detects slower-growing, less aggressive cancers because they remain detectable for longer than fast-growing cancers. For example, a slow-growing cancer that remains in Stage I for 5 years is much more likely to be found by screening than an aggressive cancer that progresses to symptoms within months. This can make screened patients appear to have better survival simply because screening preferentially found the less aggressive cancers. Overdiagnosis: Screening detects cancers that would never have caused symptoms or death during a person's lifetime, leading to unnecessary diagnosis and treatment. For example, a screening test may detect a very slow-growing prostate or thyroid cancer in an older adult that would never have become clinically important if it had remained undiscovered. Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin - LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro (03:39) - The Claim: Not Ready for Primetime (03:58) - Trial Design: 142,000 Participants (07:50) - The Primary Outcome Problem (20:29) - The Primary Endpoint: Complete Miss (22:14) - Three Arguments for the Defense (28:29) - - Statistical Sleuthing: Missing Cancers (41:14) - - The Stage Shift Argument (50:30) - - Rating the Claim

Does a little alcohol really make you speak a foreign language better? This week we unpack a quirky randomized trial that tested Dutch pronunciation after a modest buzz—and came to the opposite conclusion the researchers expected. We use it as the perfect holiday case study: instead of arguing with Uncle Joe at the dinner table, we’ll show you how to pull apart a scientific headline using a friendly, practical checklist anyone can learn. Along the way we stress-test the study’s claims, take a quick detour into what a .04% buzz actually looks like, and run our own before-and-after experiment with two brave science journalists at the ScienceWriters2025 conference in Chicago. A holiday survival guide with vodka tonics, statistical sleuthing, and a few surprisingly smooth French phrases.Statistical topicsAlternative explanationsArithmetic consistency / GRIM testBlindingEffect size / magnitudeGeneralizability / external validityObservational studies vs. experimentsOutcome measurementPICOT frameworkPlacebo and expectancy effectsPrimary outcomes / pre-specificationRandomized controlled trialsResearch hypothesesSample size SMART frameworkStatistical significance (signal vs. noise)Transparency and trustworthinessMethodological morals“​​You don't need a PhD to read a study. Just remember, PICOT and SMART.”“A decimal point can mean the difference between life and death. Details matter.”ReferencesRenner F, Kersbergen I, Field M, Werthmann J. Dutch courage? Effects of acute alcohol consumption on self-ratings and observer ratings of foreign language skills. J Psychopharmacol. 2018;32(1):116-122. doi:10.1177/0269881117735687Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin - LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com (00:00) - Intro (03:30) - Uncle Joe and the question of alcohol (07:20) - Randomized controlled trial (10:10) - PICOT mnemonic (15:43) - Just how drunk? (21:41) - Boring non-placeb (32:29) - Kristin’s SMART mnemonic (38:15) - How big of an effect? (49:29) - Two science journalists walk into a bar (55:43) - Martini scale and wrap-up

What if a haunted house makes your date look hotter? This week we dive into the infamous Scary Bridge Study — the 1970s classic that launched a thousand pop-psych takes on fear and lust. It’s the one with the swaying bridge, pretty “research assistant,” and phone number scrawled on torn paper. The study became legend, but how sturdy were its stats? We retrace the design, redo the numbers, and see how many math errors it takes to sway a suspension bridge. Along the way we find an erotic-fiction writing exercise, Adventure Dudes choosing their own experimental groups, and snarky replicators who tried (and failed) to make fear sexy again. We wrap with what the latest research says about when fear really does boost attraction — and when it backfires spectacularly. A Halloween story of danger, desire, and unconscious sexual drive. This episode has a video version! https://www.youtube.com/watch?v=2coWoS_3460Statistical topicsArithmetic checksChi-square testConfoundersGRIM testInter-rater reliabilityMeta-analysisNegative controlRandomizationReplication Sample sizeSignal vs. noiseStatistical sleuthingSubjective measurementT-testMethodological morals“Those who don't verify their numbers dig their own statistical graves.”“Famous doesn't mean flawless.”ReferencesBrown, NJ, Heathers, JA. The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science. 2017; 8(4):363-369.Dutton DG, Aron AP. Some evidence for heightened sexual attraction under conditions of high anxiety. J Pers Soc Psychol. 1974;30(4):510-517. doi:10.1037/h0037031Foster CA, Witcher BS, Campbell WK, Green JD. Arousal and attraction: Evidence for automatic and controlled processes. J Pers Soc Psychol. 1998;74(1):86-101.Kenrick DT, Cialdini R, Linder D. Misattribution under fear-producing circumstances: Four failures to replicate. Pers Soc Psychol Bull. 1979;5(3):329-334.van der Zee T, Anaya J, Brown NJL. Statistical heartburn: an attempt to digest four pizza publications from the Cornell Food and Brand Lab. BMC Nutr. 2017;3:54. Published 2017 Jul 10. doi:10.1186/s40795-017-0167-xhttp://www.prepubmed.org/grim_test/Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin - LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro: Fear and Flirtation on a Suspension Bridge (05:40) - A Classic 1970s Experiment with No IRB to be Found (11:15) - Adventure Dudes Choose Their Own Bridge (17:00) - The Sexy Story Scale (22:20) - Cool Factor and the Negative Control (28:10) - Grim Reaper Math (35:45) - T-Tests, Chi-Squares, and Shaky Results (42:00) - Electric Shocks and Damsels in Distress (49:30) - Replications and Rejections (57:20) - Wrap-Up, Methodological Morals, and a New Sexy Rating Scale

P-values show up in almost every scientific paper, yet they’re one of the most misunderstood ideas in statistics. In this episode, we break from our usual journal-club format to unpack what a p-value really is, why researchers have fought about it for a century, and how that famous 0.05 cutoff became enshrined in science. Along the way, we share stories from our own papers—from a Nature feature that helped reshape the debate to a statistical sleuthing project that uncovered a faulty method in sports science. The result: a behind-the-scenes look at how one statistical tool has shaped the culture of science itself.Statistical topicsBayesian statisticsConfidence intervals Effect size vs. statistical significanceFisher’s conception of p-valuesFrequentist perspectiveMagnitude-Based Inference (MBI)Multiple testing / multiple comparisonsNeyman-Pearson hypothesis testing frameworkP-hackingPosterior probabilitiesPreregistration and registered reportsPrior probabilitiesP-valuesResearcher degrees of freedomSignificance thresholds (p < 0.05)Simulation-based inferenceStatistical power Statistical significanceTransparency in research Type I error (false positive)Type II error (false negative)Winner’s CurseMethodological morals“​​If p-values tell us the probability the null is true, then octopuses are psychic.”“Statistical tools don't fool us, blind faith in them does.”ReferencesNuzzo R. Scientific method: statistical errors. Nature. 2014 Feb 13;506(7487):150-2. doi: 10.1038/506150a. Nuzzo, R., 2015. Scientists perturbed by loss of stat tools to sift research fudge from fact. Scientific American, pp.16-18.Nuzzo RL. The inverse fallacy and interpreting P values. PM&R. 2015 Mar;7(3):311-4. doi: 10.1016/j.pmrj.2015.02.011. Epub 2015 Feb 25. Nuzzo, R., 2015. Probability wars. New Scientist, 225(3012), pp.38-41.Sainani KL. Putting P values in perspective. PM&R. 2009 Sep;1(9):873-7. doi: 10.1016/j.pmrj.2009.07.003.Sainani KL. Clinical versus statistical significance. PM&R. 2012 Jun;4(6):442-5. doi: 10.1016/j.pmrj.2012.04.014.McLaughlin MJ, Sainani KL. Bonferroni, Holm, and Hochberg corrections: fun names, serious changes to p values. PM&R. 2014 Jun;6(6):544-6. doi: 10.1016/j.pmrj.2014.04.006. Epub 2014 Apr 22. Sainani KL. The Problem with "Magnitude-based Inference". Med Sci Sports Exerc. 2018 Oct;50(10):2166-2176. doi: 10.1249/MSS.0000000000001645. Sainani KL, Lohse KR, Jones PR, Vickers A. Magnitude-based Inference is not Bayesian and is not a valid method of inference. Scand J Med Sci Sports. 2019 Sep;29(9):1428-1436. doi: 10.1111/sms.13491. Lohse KR, Sainani KL, Taylor JA, Butson ML, Knight EJ, Vickers AJ. Systematic review of the use of "magnitude-based inference" in sports science and medicine. PLoS One. 2020 Jun 26;15(6):e0235318. doi: 10.1371/journal.pone.0235318. Wasserstein, R.L. and Lazar, N.A., 2016. The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), pp.129-133.Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin - LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro & claim of the episode (01:00) - Why p-values matter in science (02:44) - What is a p-value? (ESP guessing game) (06:47) - Big vs. small p-values (psychic octopus example) (08:29) - Significance thresholds and the 0.05 rule (09:00) - Regina’s Nature paper on p-values (11:32) - Misconceptions about p-values (13:18) - Fisher vs. Neyman-Pearson (history & feud) (16:26) - Botox analogy and type I vs. type II errors (19:41) - Dating app analogies for false positives/negatives (22:02) - How the 0.05 cutoff got enshrined (23:46) - Misinterpretations: statistical vs. practical significance (25:22) - Effect size, sample size, and “statistically discernible” (25:51) - P-hacking and researcher degrees of freedom (28:52) - Transparency, preregistration, and open science (29:58) - The 0.05 cutoff trap (p = 0.049 vs 0.051) (30:24) - The biggest misinterpretation: what p-values actually mean (32:35) - Paul the psychic octopus (worked example) (35:05) - Why Bayesian statistics differ (38:55) - Why aren’t we all Bayesian? (probability wars) (40:11) - The ASA p-value statement (behind the scenes) (42:22) - Key principles from the ASA white paper (43:21) - Wrapping up Regina’s paper (44:39) - Kristin’s paper on sports science (MBI) (47:16) - What MBI is and how it spread (49:49) - How Kristin got pulled in (Christie Aschwanden & FiveThirtyEight) (53:11) - Critiques of MBI and “Bayesian monster” rebuttal (55:20) - Spreadsheet autopsies (Welsh & Knight) (57:11) - Cherry juice example (why MBI misleads) (59:28) - Rebuttals and smoke & mirrors from MBI advocates (01:02:01) - Winner’s Curse and small samples (01:02:44) - Twitter fights & “establishment statistician” (01:05:02) - Cult-like following & Matrix red pill analogy (01:07:12) - Wrap-up

Are we all secretly ageist when it comes to dating? We put the stereotype that older men prefer younger women under the microscope using data from thousands of blind dates. What we found surprised us: the “age penalty” was real but microscopic, women wanted younger partners too, and hard age cutoffs weren’t so hard after all. Along the way, we unpack statistical significance versus practical importance, play with the infamous “half your age plus seven” rule, and imagine what it would take for love to die out… somewhere around age 628.Statistical topicsDiscontinuous regressionEffect sizesExtrapolation pitfallsLinear regressionLogistic regressionOdds ratiosOpen dataStatistical significance vs. practical significanceMethodological morals“Do not be swept off your feet by statistical significance. Tiny effects in bed are still tiny.”“Fancy units sound smart, but plain English wins hearts.”Show Notes Technical Appendix (with step-by-step explanations)ReferencesEastwick PW, Finkel EJ, Meza EM, Ammerman K. No gender differences in attraction to young partners: A study of 4500 blind dates. Proc Natl Acad Sci U S A. 2025 Feb 4;122(5):e2416984122. Matchmaking Dataset and Code on Open Science Framework: https://osf.io/rkm2d/?view_only=a0fe91dae0464077af7772e6890a8151Nuzzo RL. Communicating measures of relative risk in plain English. PM&R. 2022 Feb;14(2):283-7.O'Rell, Max. Her Royal Highness, Woman: And His Majesty--Cupid. Abbey Press, 1901.Sainani KL. Logistic regression. PM&R. 2014 Dec;6(12):1157-62.Sainani KL. Understanding odds ratios. PM&R. 2011 Mar;3:263-7. Sainani KL. Clinical versus statistical significance. PM&R. 2012 Jun;4:442-5.Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin - LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro (04:01) - Half-your-age-plus-seven rule (09:15) - Matchmaking service for the study (17:05) - Blind dates as natural experiments (21:55) - Regression results part 1: Age penalties? (28:38) - Wait, how big of an effect was that? (34:09) - Odds ratio of a second date (38:01) - Surprising age pair-ups (40:53) - Regression results part 2: Deal-breaking age limits? (44:27) - Why the patterns may or may not be true (46:30) - Wrap-up, ratings, and methodological morals

Can correcting misinformation make it worse? The “backfire effect” claims that debunking myths can actually make false beliefs stronger. We dig into the evidence — from ghost studies to headline-making experiments — to see if this psychological plot twist really holds up. Along the way, we unpack interaction effects, randomization red flags, and what happens when bad citations take on a life of their own. Plus: dirty talk analogies, statistical sleuthing, and why “familiarity” might be your brain’s sneakiest trick.Statistical topicsComputational replicationReplicationBlock randomizationProblems in randomizationBad citingInteractions in regressionUnpublished "Ghost Paper"PDF retrieved from the Wayback MachineCitationsNyhan B, Reifler J. When corrections fail: The persistence of political misperceptions. Political Behavior. 2010;32:303–330.Skurnik I, Yoon C, Schwarz N. “Myths & Facts” about the flu: Health education campaigns can reduce vaccination intentions. Unpublished manuscript, PDF posted separately.Schwarz N, Sanna LJ, Skurnik I, et al. Metacognitive experiences and the intricacies of setting people straight: Implications for debiasing and public information campaigns. Advances in Experimental Social Psychology. 2007;39:127–61.Lewandowsky S, Ecker UKH, Seifert CM, et al. Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest. 2012;13:106–131.Pluviano S, Watt C, Della Sala S. Misinformation lingers in memory: Failure of three pro-vaccination strategies. PLOS ONE. 2017;12:e0181640.Pluviano S, Watt C, Ragazzini G, et al. Parents’ beliefs in misinformation about vaccines are strengthened by pro‑vaccine campaigns. Cognitive Processing. 2019;20:325–31.Wood T, Porter E. The elusive backfire effect: Mass attitudes’ steadfast factual adherence. Political Behavior. 2019;41:135–63.Nyhan B, Porter E, Reifler J, Wood TJ. Taking fact-checks literally but not seriously? The effects of journalistic fact-checking on factual beliefs and candidate favorability. Political Behavior. 2020;42:939–60.Ecker UKH, Hogan JL, Lewandowsky S. Reminders and repetition of misinformation: Helping or hindering its retraction? Journal of Applied Research in Memory and Cognition. 2017;6:185–92.Swire B, Ecker UKH, Lewandowsky S. The role of familiarity in correcting inaccurate information. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2017;43:1948–61.Ecker UKH, O’Donnell M, Ang LC, et al. The effectiveness of short- and long-format retractions on misinformation belief and recall. British Journal of Psychology. 2020;111:36–54.Ecker UKH, Sharkey CXM, Swire-Thompson B. Correcting vaccine misinformation: A failure to replicate familiarity or fear-driven backfire effects. PLOS ONE. 2023;18:e0281140.Cook J, Lewandowsky S. The Debunking Handbook. University of Queensland. 2011.Lewandowsky S, Cook J, Ecker UKH, et al. The Debunking Handbook 2020. Available at https://sks.to/db2020. Swire‑Thompson B, DeGutis J, Lazer D. Searching for the backfire effect: Measurement and design considerations. Journal of Applied Research in Memory and Cognition. 2020;9:286–99.Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin - LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - (00:00) - Intro (02:05) - What is the backfire effect? (03:55) - The 2010 paper that panicked fact-checkers (06:25) - The ghost paper what it really said (12:35) - Study design of the 2010 paper (18:25) - Results of the 2010 paper (19:55) - Crossover interactions, regression models, and intimate talk (25:24) - Missing data and cleaning your bedroom analogy (28:11) - Fact-checking the fact-checking paper (33:07) - Replication and pushing the data to the limit (36:59) - The purported backfire effect spreads (41:06) - The 2017 paper that got a lot of attention (44:25) - Statistical sleuthing the 2017 paper (48:51) - Will researchers double down on their earlier conclusions? (54:46) - A review paper sums it all up (56:00) - Wrap up, rating, and methodological morals