All articlesSecurity awareness

Phishing simulations: what to measure and why

Phishing simulations: what to measure and why

In short

  • A simulation measures behaviour; it is not a test to catch people out.
  • Model pretexts on real campaigns, such as fake mail administrator messages about full mailboxes.
  • Report rate and time to report matter more than click rate.
  • After a click, give immediate feedback and targeted training, never public shaming.
  • Results count as evidence for ISO 27001 control 6.3 and ENS measures mp.per.3 and mp.per.4.

What is a phishing simulation and how do you measure it?

A phishing simulation sends employees realistic but harmless fake phishing emails to measure how they respond. The most useful metrics are how many people report the email, how quickly they report it, and who clicks repeatedly, not the click rate alone. Results guide targeted training and provide evidence for ISO 27001 control 6.3 on awareness and training.

What a phishing simulation is, and what it isn't

A phishing simulation is a controlled exercise in which an organisation sends its own employees emails designed to look like real phishing, then records what each person does: ignores it, reports it, clicks the link, opens the attachment or enters credentials on a fake page. Nothing harmful happens. The link leads to a page explaining what the person missed, and the result goes into a report.

The purpose is to measure and improve behaviour, not to catch people out. A simulation answers questions that training alone cannot: would people in this organisation recognise a convincing phishing email today, would they report it, and how fast would the security team find out? Those answers decide where training effort goes next.

It is also worth being clear about what a simulation is not. It is not a substitute for technical controls such as email filtering, multi factor authentication and endpoint protection, which stop most phishing before anyone sees it. It is not a one off event that proves the organisation is safe. And it is not a disciplinary tool. Programmes run as a way to punish people produce fewer reports, not fewer clicks, because employees learn to keep quiet about mistakes.

How to design a realistic pretext

The pretext is the story the email tells: why the recipient should click. A simulation built on an obviously fake email, with poor grammar and an unknown sender, measures nothing useful. Real attackers increasingly write clean, plausible messages that fit the recipient's working day. The simulation has to be at least as good.

Mirroring real campaigns

The best source of pretexts is what attackers actually send. Spain's national cybersecurity institute, INCIBE, has warned businesses repeatedly about campaigns that impersonate the company's own mail administrator. Variations include a message claiming the mailbox is full and must be updated to keep receiving email, one claiming the password is about to expire, and one claiming the webmail service, such as Office 365 or Outlook, is being upgraded and the user must log in again. The aim in every case is to capture the user's email credentials on a fake login page.

These are excellent models for a simulation because they target something every employee has, use urgency without being absurd, and lead to the outcome that matters most: credential theft. Other realistic pretexts include a shared document notification, an invoice from a known supplier, a parcel delivery notice, a message that appears to come from HR about a policy change, or a request that seems to come from a senior manager. The organisation's own incident history is also a source: if a real phishing email reached employees last month, a version of it makes a strong simulation.

Difficulty levels and segmentation by role

Not every email should be equally hard. A programme typically mixes easier messages, which most people should spot, with harder ones that test attention to detail. Tracking results by difficulty shows whether the organisation is improving or simply getting used to one style of message.

Segmenting by role makes simulations more realistic and more useful. Finance teams receive invoice and payment change pretexts; people with administrator rights receive messages targeting privileged access; executives receive messages that imitate board communications or legal notices. A generic message sent to everyone underestimates the risk for the roles attackers target most.

How often to run simulations

Frequency is a balance between keeping awareness high and causing fatigue. A single annual campaign gives a snapshot but little improvement; people forget within months. Many organisations run a campaign every month or every quarter, sending different messages to different people at different times so the exercise does not become a recognisable event that colleagues warn each other about.

Spreading sends over days or weeks also produces better data. If everyone receives the same email at nine on Monday morning, the first person to spot it often warns the rest, and the results measure office communication rather than individual judgement. Randomised timing and varied templates give a more honest picture.

Which metrics actually matter

Most simulation platforms report many numbers. Only a few of them tell you whether the organisation is getting safer, and the most commonly quoted one, click rate, is not the most important.

Report rate and time to report

The report rate is the share of recipients who report the email through the agreed channel, typically a report button in the mail client. It measures the behaviour that actually protects the organisation: when a real phishing email arrives, someone who reports it gives the security team the chance to remove it from every other inbox before more people interact with it.

Time to report measures how quickly the first reports arrive. In a real attack, the first report starts the response: the security team can search for the same message across all mailboxes, block the sender and the link, and reset any credentials already entered. An organisation where the first report arrives within minutes is in a very different position from one where it arrives the next day.

Repeat clickers

Some people click once and learn from it. Others click repeatedly across campaigns. Repeat clickers represent a concentrated risk and a clear signal for targeted support: shorter, more frequent training, a conversation with their manager about the pressures that make them rush, or additional technical controls on their accounts.

Tracking individuals across campaigns raises data protection questions, so results should be handled with care: limited access, a clear purpose, and involvement of the data protection officer when the programme is designed. The aim is to help people improve, and the programme should be explained to staff in those terms.

Why click rate alone misleads

Click rate depends heavily on how hard the email was. An organisation can halve its click rate by sending easier messages, without anyone becoming more careful. It also blurs very different outcomes: clicking a link is not the same as entering credentials on the page that follows, and the second is the event that would hand an attacker access.

The figures themselves can also carry some noise. Depending on how email is filtered and opened in each organisation, a few recorded opens or clicks may not come from a person at all, so it is worth asking the simulation provider how its reports handle automated activity. Looking at credential submissions, reports and repeat behaviour together gives a far more reliable picture than click rate on its own.

What to do after someone clicks

The moment after a click is the most valuable teaching moment in the whole programme. The person has just made the mistake; they are paying attention. What happens next decides whether the simulation builds awareness or resentment.

Immediate feedback, no shaming

The landing page should explain, briefly and without judgement, what the person missed: the sender address that did not match, the link that pointed somewhere unexpected, the urgency that should have prompted a pause. A short example of what to check next time is more useful than a long warning.

Results should never be shared publicly with names attached, and managers should not use them to embarrass anyone. People who feel ashamed after clicking a simulation are less likely to report a real phishing email they have already clicked, which is exactly the moment the security team most needs to know.

Targeted training

Training is most effective when it follows directly from what someone did. A person who entered credentials needs to learn how to check a login page and why multi factor authentication matters; a person who opened an attachment needs to learn about document based attacks. Short modules assigned automatically after a failed simulation tend to work better than a single annual course for everyone.

Organisation wide trends point to broader fixes. If many people fall for the same pretext, the training content may be missing something, or a technical control may need tightening. Simulation results should feed the awareness programme, not just measure it.

How simulations count as evidence for ISO 27001 and ENS

Neither ISO 27001 nor ENS requires phishing simulations by name. Both require organisations to make staff aware of security and train them, and both expect evidence that this happens and works. A well run simulation programme is one of the most convincing forms of that evidence, because it measures behaviour rather than course completion.

ISO 27001 control 6.3

Control 6.3 in Annex A of ISO 27001:2022, information security awareness, education and training, expects personnel to receive appropriate awareness, education and training, with regular updates relevant to their role. Clause 7.3 of the standard, awareness, adds that people working under the organisation's control must be aware of the security policy, their contribution and the implications of not conforming.

Auditors typically look for a planned awareness programme, records of who received what, and evidence that its effectiveness is assessed. Simulation results over time, together with the follow up training they triggered, show both. The security awareness for ISO 27001 post covers the wider programme this sits within.

ENS: awareness and training measures

For readers outside Spain: the Esquema Nacional de Seguridad (ENS), set out in Royal Decree 311/2022, applies to the Spanish public sector and private companies that provide services to it. Article 15 requires that staff, own or external, are trained and informed of their security duties, and Annex II includes two specific measures, mp.per.3 (awareness) and mp.per.4 (training), both applicable from the BASIC category upwards.

Annex II also includes mp.s.1, protection of email, which covers the technical side of the same threat. For companies supplying the public sector, simulation results provide concrete evidence for mp.per.3 and mp.per.4 at audit, alongside the training records themselves.

How simulations fit with email security and device controls

Simulations work best as one layer of several. Email filtering and authentication standards such as SPF, DKIM and DMARC reduce how much phishing reaches inboxes in the first place. Multi factor authentication limits what stolen credentials can do. Endpoint protection and device management, covered in the MDM post, contain what happens if a malicious attachment is opened.

The report button connects all of them. Every report of a suspicious email, simulated or real, should reach the people who can act on it: search for the same message, remove it, block the source and check whether anyone entered credentials. A simulation programme that trains people to report, combined with a team that responds to every report, turns employees into an early warning system rather than the weakest link.

Qalea runs phishing simulations and awareness training as part of its People area, using campaigns to measure the team's actual response and identify who needs more training, with results kept as evidence for the frameworks the organisation is certified against. The security awareness topic page collects the rest of the series, and the pentest vs vulnerability scan post covers the technical side of testing defences.

Frequently asked questions

How often should phishing simulations be run?

There is no fixed rule. Many organisations run phishing simulations monthly or quarterly, sending different emails to different people at different times. A single annual campaign gives a snapshot but rarely changes behaviour, while very frequent identical campaigns cause fatigue. Varied templates and randomised timing produce the most reliable results.

How do you set a baseline for phishing simulations?

Run a first campaign with a realistic, moderately difficult email before any targeted training, and record report rate, time to first report, credential submissions and click rate. That baseline is the reference for later campaigns. Compare like with like: results only show progress when difficulty stays comparable.

Do phishing simulations count as security awareness training for ISO 27001?

Phishing simulations support ISO 27001 control 6.3 on awareness, education and training, but they do not replace a training programme. Auditors look for planned training, records of who received it, and evidence of effectiveness. Simulation results, with the follow up training they trigger, are strong evidence of effectiveness.

Should employees be told in advance about phishing simulations?

Employees should know that the organisation runs a phishing simulation programme, why it exists and how results are used, but not the dates or content of individual campaigns. Announcing a specific campaign removes its value. Involving the data protection officer when designing the programme helps define how results are handled.

More articles

All articles