Past the vanity metrics: measuring security that actually matters

A security leader recently shared a thought on LinkedIn that has stayed with me: “Most security dashboards look impressive. Green metrics. Clean charts. Confident reports. But one quiet question often reveals the truth. Are we measuring activity, or measuring real protection?”

It’s a question almost every security programme should be uncomfortable answering honestly. The dashboards we present to boards, audit committees, and ourselves are often impressive—but they too frequently describe what we do rather than whether we are safer. And in a compliance landscape that increasingly demands evidence of effectiveness rather than just evidence of activity, that gap is becoming material.

This piece is about the practical question of what to measure—and what changing some of those measurements could look like for organisations approaching ISO 27001 certification, NIS2 obligations, or simply trying to build a security function that earns its budget.

Why activity metrics dominate

Activity metrics dominate security reporting for understandable reasons. They are:

Cheap to collect — most tools produce activity output by default
Easy to present — activity rises and falls in ways charts handle well
Politically safe — reporting “we ran 47 training campaigns” is harder to challenge than reporting “we reduced phishing susceptibility by 8%”
Compatible with audit narratives — an external auditor can tick a control box if you show evidence of activity, even if the activity isn’t accomplishing anything

What activity metrics rarely tell you is whether your organisation is actually safer. They confuse motion with progress.

The classic examples almost write themselves.

Training completion rate. 98% of staff completed annual security awareness training. Useful as a compliance signal; mostly silent on whether anyone retained anything or behaves differently.

Patches deployed. 12,000 patches applied last month. Useful as workload data; says nothing about whether the patches were the ones that mattered, or whether the critical-rated vulnerabilities are actually closed.

Alerts triaged. 5,000 alerts handled this quarter. Useful as SOC capacity data; says nothing about whether the things you should have detected were actually detected.

These metrics are not wrong, but they are not sufficient. They describe a system that is functioning. They do not describe a system that is protecting.

What it costs to measure the wrong things

There are real consequences of building a security narrative on activity metrics alone.

Mis-prioritised investment. When the metrics reported upwards are activity-based, the activities that get funded tend to be those that produce visible activity. Volume of tickets handled, coverage of tools, number of trainings delivered. Resilience work—which produces nothing visible until it is needed—gets short-changed.

Mis-calibrated assurance. Boards reading green dashboards form a confidence that may not be warranted. The first time the confidence is tested is during an incident, and by then it’s too late to recalibrate.

Mis-aligned compliance. External certification bodies have been getting better at distinguishing controls that are documented and operating from controls that are documented and effective. ISO 27001’s Clause 9 and Annex A.5.36 both require evidence of effectiveness—not just evidence that the activity occurred. Auditors who scratch beyond the surface increasingly find a gap between the documented dashboard and the operational reality.

Mis-recognised people. The teams who produce activity metrics tend to get credit. The teams who quietly reduce risk tend to be invisible until something fails. Over time this shapes the capabilities the organisation hires for and retains.

What good security metrics actually measure

The alternative to activity metrics isn’t fewer metrics—it’s better-chosen ones. A meaningful security dashboard answers five outcome-level questions about the organisation.

1. Can we operate when challenged?

This is the resilience question. The metrics that answer it test recovery capability under stress, not capability on paper:

Recovery time objective (RTO) compliance, measured by actual recovery exercises—not aspirational targets
Disaster recovery test success rate, including the percentage of critical systems successfully restored within target windows
Time to operate manually, for organisations with critical processes that have a degraded fallback
Backup restoration success rate, with backups actually restored under realistic conditions, not just validated by the backup system itself

If you cannot honestly answer “yes” to a credible recovery scenario, no other metric matters as much as fixing that gap.

2. How exposed are we today?

This is the risk-state question. Useful metrics describe current exposure, not historical activity:

Critical vulnerability open count and age, weighted by exploitability and asset criticality—not raw count
External attack surface trend, measured by inventory of internet-exposed services and their hardening
Privileged accounts without recent review or recent activity — often more revealing than total account count
Third-party risk concentration: how many critical processes depend on the same supplier or platform

These metrics tend to be uncomfortable. They show real exposure rather than aspirational coverage. That’s the point.

3. Are we getting better?

This is the trajectory question. Movement over time is the most honest signal in security reporting:

Mean time to detect (MTTD) and mean time to respond (MTTR), with quarterly trend lines rather than point-in-time figures
Audit and assessment finding closure, with age and severity distribution—not just count
Repeat findings: issues identified in past reviews that recur in subsequent ones. The single best signal of management system health.
Control failures discovered in operational testing, with closure rate over time

The shape of these curves across twelve months tells you more about whether your programme is improving than any point-in-time figure.

4. Are people part of the defence?

Human factors are where most volume breaches originate. As the Irish DPC’s 2024 annual report makes clear, roughly half of all breaches notified to the regulator involved correspondence going to the wrong recipient. Useful human-factor metrics test whether your people are detecting threats and behaving safely, not whether they completed training:

Phishing simulation click rate, trend, with severity-weighted scenarios
Phishing simulation report rate (vs click rate). Click-rate-going-down is good. Report-rate-going-up is better.
Insider-error breach rate, by department and process—because mitigation usually involves systems, not lectures
Privileged user behaviour anomalies detected and investigated

A well-designed human factor programme treats employees as detection assets, not as failure points to be drilled into compliance.

5. Are we paying attention to the boring things?

Most preventable security incidents are not exotic. They occur because something dull was left undone: an account not removed, a control not reviewed, a log source not connected. Boring metrics catch boring failures before they matter:

Privileged access review completion, by required cadence
MFA coverage, particularly for legacy systems and external-facing applications
Log source coverage of critical systems, with gaps actively tracked
Joiner-mover-leaver process completion, with measured cycle times

If anything, these metrics are the strongest predictor of whether a security programme is being run as a discipline or as a series of projects.

What standards expect (whether they say so directly or not)

The shift from activity to effectiveness is now embedded in major compliance frameworks, even where the standards don’t use those exact words.

ISO 27001’s Clause 9 (Performance evaluation) requires you to determine what needs to be monitored and measured, the methods to ensure valid results, when monitoring happens, and who analyses the results. A management system that monitors only activity—and never asks whether the activity is producing intended outcomes—is technically non-conforming with this clause. In practice, this is one of the clauses where audit findings are increasing.

NIS2’s Article 21 requires policies and procedures to assess the effectiveness of cybersecurity risk management measures as one of its ten minimum measures. Effectiveness is the operative word. Competent authorities will increasingly look for evidence that organisations have honestly assessed whether their controls work—and have changed them when they don’t.

ISO 22301’s Clause 9 mirrors ISO 27001 for business continuity, with particular emphasis on the assessment of business impact and the validity of recovery testing. The bias toward outcome metrics is stronger here because the consequences of unreal capability are usually more immediate.

In all three cases, the auditor’s question is the same: “How do you know this is working?” If the answer is a list of activities rather than evidence of outcomes, the control is exposed regardless of how well it is documented.

A practical transition

Most security programmes can’t replace their existing dashboard overnight—and shouldn’t try. The metrics in place often serve real audiences (audit committees, regulators, certification bodies) who would be alarmed by a sudden discontinuity. The shift is best made incrementally.

A reasonable starting position:

Pick one outcome-level metric per domain. Don’t try to redesign everything. Start with one: the resilience question, the exposure question, the trajectory question, the human-factor question, the attention-to-detail question. One each.
Run both old and new in parallel for a quarter. This lets you build confidence in the new metric, calibrate thresholds, and prepare audiences for the shift in narrative.
Be honest with boards about the change. A move from green dashboards to mixed-colour outcome dashboards needs to be framed as increased rigour, not as a deterioration. Most boards welcome the change once they understand it.
Retire activity metrics as outcomes mature. Some activity metrics will still be appropriate to retain—training completion remains a useful compliance signal, for instance. Others can be quietly retired once outcome metrics offer more insight.
Tie metrics back to risk acceptance. Each meaningful metric should have an associated risk-tolerance threshold—not just a target. The conversation with the board is then about acceptable risk, not about cosmetic performance.

Closing thoughts

The risk with the green-dashboard mindset isn’t that the metrics are dishonest. It’s that they’re insufficient. They describe motion in the system without describing whether the system is achieving its purpose. In an environment where attackers, regulators, customers, and insurers are all converging on the question of outcomes, organisations that continue to report only activity will find themselves increasingly out of step.

As the LinkedIn post that prompted this piece concludes: cybersecurity maturity is not defined by the number of tools deployed—it is defined by the signals you monitor and the risks you understand.

The harder version of that is this. Maturity is defined by the signals you are willing to have measured honestly, even when they come back red.