How to Pass Every Audit - Measure

Audit
Written By Andrew Alaniz

Your scope is everything when it comes to metrics. Your metrics will show that you are governing properly. A metric that is always 100% can often indicate a denominator problem rather than a perfect program. This is a big reason to have multiple measurements. In fact, I would advise defining your metrics to have a threshold of tolerance. Things change, systems move, are removed, are added, and there are processes that have to happen in order to finalize systems, and your metrics should account for that. I will add a caveat here by saying that the amount of data that you have available to you greatly determines the accuracy of your metrics. If you are unable to see where a system is within its lifecycle, it greatly diminishes your ability to add precision to your metrics.

You want to define your metrics within your standards. For the most part (not universal but close) each standard statement should have a metric tied to it. A metric also does not have to stop at a single measurement either. You will want measurements that are available to you as a team and then measurements that are available to your executives and risk committees. This is not to say you should hide anything but clearly define where the risk is. We will continue our MFA example.

Some measurements that you probably want to have for your MFA standard:

MFA Metrics Dashboard
Human Non-human Total M0 — Total managed accounts Validates human + non-human scope coverage 9,760 human accounts ~25,600 non-human accounts ~35,360 total accounts Employees Contractors Third parties M1 — Total human accounts Full landscape view 9,280 employees 453 contractors 27 third parties Total: 9,760 M2 — Active accounts Accounts that exist but are disabled 98% — 9,094 active employees 80% — 362 active contractors 44% — 12 active third parties Active: 9,468 292 inactive M3 — Active with at least one login Filters accounts in setup/provisioning 98% — 8,912 logged in employees 98% — 354 logged in contractors 100% — 12 logged in third parties Total: 9,278 M4 — MFA enforced Program effectiveness — exec/risk committee metric 100% — 8,912 employees + MFA 99.7% — 353 1 account — no MFA 100% — 12 third parties + MFA Total: 9,277 1 acct no MFA out of scope boundary — non-human identities M5 — Non-human accounts with interactive login enabled Out of scope — monitored 0.5% — 128 of ~25,600 non-human accounts Non-human identities with interactive login enabled. Governed under separate standard. ~25,600 M6 — Non-human with interactive login enabled and used Action required 1 account — actively used Non-human identity with interactive login in active use. Requires investigation and remediation. Color guide: Healthy Needs attention Action required Executive metric (M4) Landscape / context only

These are purely made-up numbers, but what you can see are a number of metrics that you can use to show audit a few things.

M0 and M1 show audit that you are watching your scope. M5 and M6 are showing audit you are watching your out of scope to ensure that it shouldn’t be in scope. M2 and M3 are accounting for account lifecycles and giving you a threshold for accuracy. M4 is the only metric that needs to be shared proactively outside of your team. Whether it makes it to board metrics or risk committee is a leadership call, but the others are your view of your program’s world. And you can use that view to zoom in on the real risk.

You may still have some auditors that want to pick these apart. If they are automated, they may ask to validate calculations. The way they would do that is to query the systems of record themselves and compare their calcs to yours. You may have to in turn audit their calcs if they are off to ensure you are each calculating correctly.

Another common validation they will use is a test of 1. They will pick a random account and validate that it has MFA. Two risks come with this approach. One is that they pull a non-human or disabled account, just be prepared to explain. They will pick another data point. The other risk is that they pick the one account that is using interactive login, and is out of compliance. What you will want here is a documented process for how you are notified if this ever happens, how you track the exception, your process for remediating, and your evidence for all of this. As you can see metrics can get unwieldy quickly, but they are essential to move you from reactive auditing to continuous compliance.

If you can automate these metrics, then you can give audit access to the metrics and exception tracking and they can audit whenever they want to. Keep in mind that even with solid metrics, there will always be accounts that don't comply for legitimate reasons. That's what an exception process is for which is covered in the next post.

Andrew Alaniz https://www.alaniz.io
Previous

How to Pass Every Audit - Exception Process
Next

How to Pass Every Audit - Out of Scope