Randomised or observational? Before comparing quit rates, ask who chose

01

Recalculate an invented headline

This is invented arithmetic, not evidence about any SMS service. All adults in an imagined service have the same starting information and may choose optional support messages. Nobody is randomly allocated. Before that choice, some have a plan prepared and others are still preparing one. These categories and all the counts below are created for this exercise, not claims about actual readiness or quit rates.

At three months, the same question asks everyone about no cigarette smoking in the preceding seven days. Assume every answer is known. Among message users, 10 of 50 with a prepared plan and 15 of 150 still preparing meet that outcome: 25/200 = 12.5%. Among non-users, the counts are 30/150 and 5/50: 35/200 = 17.5%. A headline might say users did five percentage points worse.

Now read within each starting category: 10/50 and 30/150 are both 20%; 15/150 and 5/50 are both 10%. The overall difference reflects the different mixtures in these invented counts. Users have more people from the 10% category. That explains this arithmetic; it neither proves that real messages are harmful nor establishes no causal effect, even in this hypothetical comparison.

A careful summary of our exercise is ‘Self-selected users had a lower overall observed proportion; the groups' starting composition differed.’ It must not become ‘Messages reduced quitting’. We supplied no randomised outcomes, uncertainty intervals or real-service evidence. Compare public methods and aggregate tables only; no reader's smoking history or health information is needed.

02

What a real adjustment would still need

Cochrane defines confounding as common causes of intervention choice and the outcome. Recorded starting characteristics can be used in a suitable analysis, but unrecorded factors, measurement error and model assumptions remain concerns. Looking at one category does not certify that all confounding is gone. Adjusting for something changed by the intervention can itself introduce bias.

For a real study, ask which relevant factors were measured before use, how they were measured and why the chosen method can address them. Also inspect when follow-up starts and whether users and non-users are assessed in the same way. Do not replace that reasoning with ‘large database’ or ‘adjusted result’. The direction of bias is not fixed: a different selection pattern could favour users instead.

[2]

03

Change the allocation process, not the observed result

An alternative trial could randomly assign the offer of messages, with future assignments concealed until enrolment is confirmed. It would compare assigned offers rather than self-selected users. On average, properly implemented randomisation prevents starting prognostic factors from determining assignment, including factors the researchers did not measure. It does not ensure exact balance in every realised sample; chance imbalance is not automatically evidence of a faulty process.

Concealment before assignment is different from blinding afterwards. Alternating admissions or using birth dates is not a substitute for an unpredictable random sequence. Read what was actually done rather than accepting the word ‘randomised’ alone.

Later missing outcomes, different measurement or selected reporting can still undermine a randomised result. Simply regrouping people by actual message use, or excluding those who did not use their assigned offer, can reintroduce selection. An assignment effect and an adherence effect are distinct questions; appropriate analysis and missing-data assumptions still matter.

[1]

04

Do not discard observation when it answers another question

Non-randomised studies can address questions not fully covered by trials, including longer follow-up, rare outcomes or different settings. Their ability to support causal inference depends on design features and assumptions, not just a cohort label. Describing who accesses a service is useful too, but it is not the same task as estimating what the service causes.

[3]

05

For personal support, leave the arithmetic behind

NHS guidance lists routes to professional stop-smoking support across the UK, with local differences. Elsewhere, consult qualified local help. Neither this invented table nor a study-design label chooses your support or treatment; those questions belong in an appropriate professional conversation.

[4]

06

What to keep in mind

• First identify how people entered the comparison groups.
• A matched proportion in one starting category is not proof of no causal effect.
• Randomisation protects assignment, not every later analysis.

07

Common questions