1) Executive Summary

This guide turns research papers into readable maps. You’ll learn where truth tends to hide (study design), how certainty is built (methods, bias control, power), and how to translate an effect into a decision you can live with (absolute risk, NNT/NNH, value trade-offs). It’s for builders, carers, and operators who need actionable reading skills—not academic mystique.

2) Evidence Hierarchy

Evidence is not flat. Some designs answer causal questions better; others are for signal-finding. A sensible hierarchy (for causal effectiveness) is:

2.1 The Pyramid (for Causality)

• Systematic Reviews & Meta-analyses of High-Quality RCTs: Synthesis of similar, well-run trials with transparent methods. Inspect heterogeneity and risk of bias.
• Randomised Controlled Trials (RCTs): Randomisation neutralises many confounders. Power, blinding, allocation concealment, and pre-registration matter.
• Prospective Cohorts: Observe exposure before outcome; good for associations, less robust for causality (confounding).
• Case-Control: Efficient for rare outcomes; sensitive to recall and selection bias.
• Cross-Sectional: Snapshot; great for prevalence, poor for causality (temporality missing).
• Case Series/Reports: Hypothesis generators; beware base-rate neglect.
• Mechanistic/Bench/Animal: Useful for plausibility; alone, can mislead about real-world effect size and safety.

2.2 Don’t Worship the Pyramid

Hierarchies guide, they don’t dictate. A tiny, biased RCT can be worse than a massive, careful cohort. A meta-analysis of mismatched or low-quality trials can look precise while being wrong. Always couple hierarchy with quality appraisal.

2.3 Questions → Designs (Quick Map)

2.4 Pre-registration vs. p-Hacking

Preregistration (public protocol before data peeking) reduces cherry-picking. In its absence, watch for: “outcome switching,” “garden of forking paths,” and a blizzard of subgroup analyses without correction. A single neat p-value is not evidence that the research plan was clean.

2.5 DAG Intuition (Causality on a Napkin)

Directed Acyclic Graphs (DAGs) help decide what to adjust for. Adjusting the wrong node (like a collider) can add bias. You don’t need to draw perfect graphs—just sketch the story: what causes what, and which paths you must block to isolate the effect.

Exposure (X) ──▶ Outcome (Y)
   ▲              ▲
   │              │
Confounder (C) ───┘   → Adjust for C
Collider (K) ◀─ X ─▶ Y (do NOT adjust for K)
      

3) Methods & Bias (Setup)

Bias is not an insult; it’s a directional error built into the architecture. Good methods are anti-bias engineering. Use this checklist before you touch the results section:

3.1 Trial Hygiene (for RCTs)

• Random sequence generation: Truly random? (e.g., computer RNG, not alternation or DOB)
• Allocation concealment: Could recruiters guess next assignment?
• Blinding: Participants, clinicians, and outcome assessors?
• Pre-specified outcomes: Protocol/registry matches the paper?
• Sample size & power: Was the study powered for its primary endpoint?
• Attrition: ITT (intention-to-treat) vs per-protocol; missing data handling sensible?

3.2 Observational Hygiene

• Sampling frame: Who was included and who was reachable?
• Measurement: Were exposure/outcome measured the same way across groups?
• Confounding control: A priori adjustment plan? Propensity scores? Sensitivity analyses?
• Temporality: Did exposure clearly precede outcome?

3.3 Publication & Reporting Bias

Positive and tidy gets published; messy and null often doesn’t. Look for trial registries, grey literature, and funnel plots in syntheses. If only “winners” show up, the average effect will be inflated.

What’s Next (Part 2)

We’ll cover the stats you actually need (effect sizes, CIs, absolute vs relative risk, NNT/NNH), a practical GRADE primer, and step-by-step decision rules with mini-cases.

Tip: Bookmark this page. The full series stays at the same URL slug for consistency.

4) Stats You Actually Need

Forget the swamp of Greek letters. These are the handful of statistics that truly matter when reading a study: effect size, confidence interval, absolute vs relative risk, and NNT/NNH.

4.1 Effect Size

The effect size answers: How big is the difference? Examples include risk ratios, odds ratios, mean differences, or hazard ratios. A p-value may tell you “there is some difference,” but effect size tells you whether the difference matters.

A drug that lowers blood pressure by 1 mmHg in 10,000 people is “statistically significant” in a huge trial—but clinically trivial.

4.2 Confidence Interval (CI)

A 95% CI is the range of values consistent with the data if the study were repeated many times. Wide CI = less certainty. Narrow CI = more certainty.

• If the CI crosses “no effect” (e.g., RR=1, difference=0), the result is inconclusive.
• The width matters as much as the point estimate.

4.3 Absolute vs Relative Risk

Relative risk (RR) says “50% reduction.” Absolute risk (AR) says “2 in 100 → 1 in 100.” Always convert to absolute numbers for decisions.

4.4 NNT / NNH

Number Needed to Treat (NNT): how many people need the treatment to prevent one bad outcome. Number Needed to Harm (NNH): how many before one adverse effect occurs.

5) Certainty & GRADE

Evidence is more than effect size. Certainty depends on how much we trust the estimate. The GRADE framework (Grading of Recommendations, Assessment, Development, and Evaluations) is the global standard.

5.1 The GRADE Levels

5.2 What Downgrades Certainty

• Risk of bias: flawed randomisation, unblinded assessors, selective reporting.
• Inconsistency: wildly different results across studies without explanation.
• Indirectness: different population, intervention, comparator, or outcome than you care about.
• Imprecision: wide CIs, few events, underpowered.
• Publication bias: missing negative or null studies.

5.3 What Upgrades Certainty

• Large effect size (e.g., RR < 0.5).
• Clear dose–response gradient.
• Plausible confounding would only reduce effect (not explain it away).

5.4 GRADE in Action

Example: Multiple RCTs show a new antihypertensive lowers stroke risk by ~20%. CIs are tight, studies well-designed. Certainty: High. In contrast, observational studies suggest coffee lowers cancer risk, but results vary and confounding is likely. Certainty: Low.

What’s Next (Part 3)

We’ll move to Replication & Meta-Analyses: how to spot when pooled evidence clarifies truth—and when it misleads. Then we’ll dive into case walkthroughs from health and economics.

6) Replication & Meta-Analyses

A single study is rarely enough. Replication—independent teams, new samples, same question—is the stress test of truth. Meta-analyses try to summarise multiple studies, but can mislead if the inputs are flawed.

6.1 Replication Crisis

In psychology and biomedicine, large replication projects found that fewer than half of landmark findings reproduced. Causes include small samples, publication bias, and analytic flexibility. Replication adds weight; failure to replicate lowers certainty dramatically.

6.2 Meta-Analysis Mechanics

• Pooling: Combines effect sizes, weighted by study size/variance.
• Forest plots: Show individual study estimates and overall pooled estimate.
• Heterogeneity: I² statistic shows how consistent the studies are. High I² = lots of variation.

6.3 When Meta-Analyses Mislead

• Garbage in, garbage out: Poor-quality or biased studies can dominate.
• Mixing apples and oranges: Pooling very different interventions or populations can produce meaningless averages.
• Publication bias: Funnel plots skewed if negative studies are missing.
• Small-study effects: Tiny trials with big effects distort pooled estimates.

6.4 Systematic Review vs Meta-Analysis

A systematic review collects and appraises studies systematically. A meta-analysis is the statistical pooling step—only meaningful if studies are comparable.

7) Case Walkthroughs

Let’s apply the tools so far. We’ll walk through a health case and an economics case.

7.1 Health Case: Vitamin D & Respiratory Infections

Early observational studies suggested people with higher vitamin D had fewer infections. But confounding was strong: healthier lifestyles correlate with both higher vitamin D and fewer infections.

• RCTs: Some showed modest reduction in colds, especially in deficient populations.
• Effect size: Absolute risk reduction ~3%, NNT ~33 for seasonal prevention.
• Certainty: Moderate; stronger for deficient groups, low for general population.
• Meta-analysis: Pooled trials with different dosing regimens; heterogeneity high. Garbage-in risk.

Lesson: Always stratify by baseline risk. Supplements may help those deficient, but pooling all groups dilutes effect.

7.2 Economics Case: Minimum Wage & Employment

Classic theory predicted higher minimum wage reduces jobs. Observational data are noisy—many confounders (economic cycle, sector shifts).

• Natural experiments: Border county studies (New Jersey vs Pennsylvania fast food) showed minimal job loss.
• Replication: Later studies mixed—some small losses, some neutral, some gains.
• Meta-analyses: Large reviews show average near zero effect, but heterogeneity across contexts.
• Certainty: Moderate—effects are context-dependent, not universal.

Lesson: For economics, replication across settings is key. Don’t extrapolate one context to all.

What’s Next (Part 4)

Next, we’ll cover Practical Decision Rules: translating study findings into everyday action, including reversibility, value trade-offs, and decision frameworks you can actually apply.

8) Practical Decision Rules

Research isn’t for worship—it’s for decisions. Here are frameworks to translate study findings into everyday action:

8.1 The 4R Rule (Risk, Reversibility, Resources, Relevance)

• Risk: What’s the baseline risk, and how much does it change?
• Reversibility: Can you undo the decision if wrong?
• Resources: What time, money, or energy cost is involved?
• Relevance: Does the study population match yours?

8.2 Minimum Viable Decision (MVD)

Instead of asking “What’s the truth?”, ask: “What’s the smallest action I can take to test this safely?” That might mean a 2-week trial, a small pilot, or monitoring one metric.

8.3 The Reversibility Principle

“If a decision is reversible, favour speed and small experiments. If irreversible, demand stronger evidence.”

8.4 Weighted Evidence Table

Create a quick evidence summary before acting:

9) Worksheets & Glossary

Use these quick-reference worksheets and glossary entries to practice study literacy daily.

Glossary (Core Terms)

Absolute Risk Reduction (ARR)

Difference in event rates between groups (e.g., 10% – 5% = 5%).

Confidence Interval (CI)

Range of values consistent with the data; narrower = more precise.

Effect Size

Magnitude of difference (risk ratio, odds ratio, mean difference).

GRADE

Framework rating certainty of evidence: High, Moderate, Low, Very Low.

NNT / NNH

Number needed to treat/harm for one additional outcome.

p-value

Probability of observing data this extreme if the null hypothesis were true. Not the probability the null is true.

Publication Bias

Distortion because positive studies more likely to be published.

What’s Next (Part 5)

Final step: the 10-Day Study Literacy Sprint—a structured program to build these skills quickly and apply them in real decisions.

10) Execution Framework: 10-Day Study Literacy Sprint

Knowledge without reps fades. This sprint is a 10-day crash course to ingrain study-reading habits. Each day has one focused task and a quick worksheet. Spend 30–45 minutes daily, and by Day 10 you’ll be producing one-page Decision Briefs.

Closing Thoughts

Reading studies is not about perfection; it’s about discipline and context. You now have the tools to extract signal from noise, frame bias as engineering, and act prudently with evidence.

Science isn’t a verdict. It’s a map. You don’t need the whole map—you need to know how to read the legends, spot the traps, and move with confidence.

Bookmark this guide. Re-run the 10-Day Sprint quarterly. Build your own Decision Brief library. Over time, you’ll think less in headlines and more in structures.