What is an RCT?
A Randomised controlled trial (RCT) is the gold standard method for testing whether an intervention actually works. The core idea is simple: take a group of people, randomly split them into two groups, give one group an intervention and the other nothing (or business as usual), then compare what happens.
Two elements make an RCT an RCT:
Randomisation — participants are assigned to groups purely by chance, not by preference, performance, or any other factor. This is what separates an RCT from a simple before/after comparison.
A control group — a group that receives no intervention (or the status quo), so you have a genuine point of comparison. Without a control group, you can't know whether any change you observe was caused by the intervention, or would have happened anyway.
RCTs are considered one of the highest levels of scientific evidence. They help establish causality — not just correlation — between an intervention and a behaviour change.
Why do RCTs matter for security?
Most security awareness programmes rely on intuition, industry benchmarks, or vendor claims to decide what to run. RCTs let you answer a different and more important question: does this specific intervention actually change behaviour in my organisation?
That's a much stronger basis for decisions — and for demonstrating impact to leadership.
How CybSafe supports RCTs
The most difficult, time-consuming, and error-prone parts of setting up an RCT — randomisation, defining control and treatment groups, and structured data collection — are handled directly by the CybSafe platform.
The key feature is the Random group assignment action in Workflows.
The Random group assignment action
When building a workflow, you can add a Random group assignment step. This automatically and randomly splits users entering the workflow into two groups:
Control group — receives no intervention from the workflow
Experimental group — receives your chosen intervention (a nudge, a module, a phishing simulation, etc.)
You can set the Control group allocation — the percentage of users assigned to the control group. The default is 50%.
This single action gives your experiment its scientific foundation: true random assignment at scale, with no manual sorting required.
What the platform handles for you
✅ Random group assignment (no manual effort, no bias)
✅ Delivery of the intervention to the experimental group only
✅ Behaviour event data collection for both groups via SebDB
✅ Trigger logic (e.g., only enrol users who haven't performed a behaviour)
What you still need to do
Define a clear research question and outcome measure before you start
Choose an appropriate intervention
Decide on a study duration
Extract and analyse the data at the end (descriptive stats at minimum; statistical testing recommended for meaningful sample sizes)
The four steps to running an RCT on CybSafe
These steps follow the BIT TEST methodology (Target, Explore, Solution, Trial) adapted for the CybSafe context.
Step 1: Define the outcome
Choose a single security behaviour to target — one that is:
Measured objectively on the platform (i.e., there is a behaviour event that captures it)
Showing room for improvement in your organisation
Not currently enforced or mandated (so you can detect a genuine change)
Express your research question clearly:
"Does [intervention] increase the rate of [behaviour] among users who are currently not performing it?"
To answer this reliably, you need to compare what happens with the intervention against what happens without it. The question therefore becomes:
"Is the rate of [behaviour] higher among users who received [intervention] than among those who received nothing?"
Your outcome measure will depend on the behaviour the platform captures:
For binary events — did the user perform the behaviour or not — use a proportion (the percentage of users in each group who performed it)
For cumulative events — how many times the behaviour occurred — a count or average per group may be more appropriate
You should define both a primary and secondary outcome before you start:
Primary outcome — the main behaviour you are measuring; the one your research question is directly about. There should only be one.
Secondary outcomes — additional behaviours you track alongside the primary one. These can provide useful supporting evidence but are not the main focus of the trial.
Step 2: Specify the behavioural context
Before choosing an intervention, ideally you understand why users aren't performing the target behaviour. If not, you can move forward with an assumption and make this part of the experiment. Common barriers include:
Lack of knowledge (they don't know how)
Impact perception (they don't think it matters)
Competing priorities or time pressure
Overconfidence (they think they already do it)
This step shapes which intervention you choose in Step 3.
Step 3: Build the intervention
Choose an intervention that directly addresses the barriers you identified. On CybSafe, your options include:
Nudges — targeted messages delivered via email or Slack, designed using behavioural science principles
Modules — learning content assigned to the experimental group
Phishing simulations — used to trigger a workflow that then delivers an intervention
For a clean RCT, test one intervention at a time. If you want to compare multiple interventions, you can run multiple experimental arms — but each should have its own control comparison.
Step 4: Test, learn, adapt
Set up your workflow with the Random group assignment action, run the experiment for your defined period, then extract behaviour event data for both groups. Compare the rate of your target behaviour between the experimental and control groups.
If your sample size is large enough, run a statistical test (e.g., chi-squared for proportions) to determine whether the difference is statistically significant — not just a chance variation.
Document what you find and use it to inform future interventions.
Putting it into practice
SB087: Reports a suspicious message
SB087: Reports a suspicious message captures whether a user reports a suspected phishing email via the platform. The underlying event is EMAIL_REPORTED.
This is an ideal behaviour for an RCT because:
It's measured objectively (the event fires when a user actively reports)
It's a high-value security behaviour with clear organisational benefit
There are known barriers and existing interventions designed to address them
It has enough data volume to detect meaningful differences
Step 1: Define the outcome
Research question:
Does sending a targeted nudge increase the rate of suspicious message reporting (SB087) among users who are currently not reporting?
To answer this, we compare reporting rates between users who received the nudge and those who received nothing. The question therefore becomes:
Is the rate of suspicious message reporting higher among users who received the nudge than among those who received nothing?
Target population: Users who received a phishing simulation but did not report it — identified via the Phishing simulation delivered but not interacted with trigger in Workflows.
Primary outcome measure: Phishing reporting rate — the proportion of users in each group who trigger an EMAIL_REPORTED event within 14 days of the intervention.
Secondary outcome measures:
SB159: Opens only a legitimate link (phishing evasion rate)
SB156: Shares sensitive information only on an approved or official website (data sharing evasion rate)
Step 2: Specify the behavioural context
Known barriers to SB087 include:
Don't know how — users aren't aware of how to report, or that the report button exists
Impact perception — users underestimate the value of reporting; it feels like it won't make a difference
Lack of security knowledge — users don't recognise suspicious messages as threats worth reporting
Step 3: Build the intervention
Two nudges are well-suited to this behaviour, targeting different barriers:
Nudge 246 (Impact perception, Facilitate, Risk): "It might not feel like a big thing to do, but it saves time and money" — addresses the perception that reporting doesn't matter
Nudge 238 (Don't know how, Compare): "You know how to contact emergency services" — uses analogy to make reporting feel intuitive and important
For a simple two-arm RCT, choose one nudge as your intervention. For a multi-arm trial, you can test both nudges against a shared control group.
Modules that support SB087 and could alternatively serve as the intervention:
Spotting fake emails (story)
Security incidents: Your role (story)
Step 4: Set up the workflow
Here is how to build this RCT in CybSafe Workflows:
Trigger: Phishing simulation delivered (and not interacted with)
Wait: 7 days (to allow baseline behaviour to be observed)
Check: Has the user reported a suspicious message (EMAIL_REPORTED) in the past 7 days?
If yes → exit workflow (no intervention needed)
If no → proceed to randomisation
Random group assignment
Control group allocation: 50%
Control group → no action
Experimental group → send Nudge 246 via email
Wait: 14 days
Data extraction: Pull EMAIL_REPORTED events for all enrolled users, split by group
What to compare at the end
At the end of the study period, compare:
Metric | Control group | Experimental group |
Users enrolled | n | n |
Users who reported (EMAIL_REPORTED) | x | y |
Reporting rate | x/n % | y/n % |
A meaningful difference in reporting rate — especially one that is statistically significant — is evidence that the nudge caused the change, not random variation or other factors.
Tips for a good RCT
Pre-register your hypothesis before running the experiment. Decide what counts as success before you see the data.
Don't change anything mid-trial. Pause any other campaigns or nudges targeting the same behaviour during the study period.
Use proportions, not raw counts. A reporting rate is more meaningful than a raw number, especially if group sizes differ.
Statistical significance matters at scale. With small sample sizes (under ~30 per group), descriptive statistics are still useful — but treat results as indicative rather than definitive.
Run one test at a time per behaviour. Overlapping experiments targeting the same behaviour make results hard to interpret.
Document everything. Record your hypothesis, design, and results. Even null results are valuable — they tell you what doesn't work.
Glossary
Term | Definition |
RCT | Randomised controlled trial — an experiment where participants are randomly assigned to intervention or control groups. |
Control group | The group that receives no intervention; the baseline for comparison. |
Experimental group | The group that receives the intervention being tested. |
Random group assignment | The CybSafe workflow action that automatically and randomly splits users into control and experimental groups. |
SebDB | Security Behaviour Database — an open-access catalogue of over 70 everyday security behaviours, each with a unique identifier (e.g., SB087), mapped to real-world risks and measured via objective platform events. |
EMAIL_REPORTED | The platform event that fires when a user reports a suspicious message |
Primary outcome | The main behaviour being measured in the trial; the one the research question is directly about. There should only be one. |
Secondary outcome | Additional behaviours tracked alongside the primary outcome to provide supporting evidence. |
Statistical significance | A measure of whether an observed difference between groups is likely due to the intervention rather than chance. |