Companies that have spent significant resources and money on managing their cyber security environment understandably want to know the results of all this expenditure. As such, it is important for Managed Security Service Providers (MSSPs) to be able to provide customers with some visibility into those results. However, results only tell you half the story. For instance, they may demonstrate that there was a breach, but, without significant forensic effort, will not necessarily provide the sequence of events or failures which led up to the compromise.
Organizations are complex and have many performance measures. Most have designated key performance indicators (KPIs) at various levels of the organization, which business management agrees are the most important metrics to monitor. They are designed to be leading indicators of business performance. Key risk indicators (KRIs) are similar in that they are leading indicators; however, rather than signal performance, they signal increased probability of events that have a negative impact on business performance. Then there are key control indicators (KCIs), that are closely related to KRIs in that they measure the effectiveness of risk controls.
Business managers use KPIs to show where things are going well or poorly and KRIs to indicate when the probability of the latter is increasing. KCIs are a measure of how well risks controls are performing. MSSPs can, and should, do the same using security data which is commonly available for most of their clients.
More on KPIs, KRIs and KCIs
You may hear these terms used interchangeably; however, they are distinctive and should be treated differently in order to make them understandable.
- Key performance indicator (KPI): Shows how the business is performing based on the goals and objectives leadership has set as well as the progress that is being made toward those goals. For security operations, this metric might be used in an effort to resolve open items or tackle a backlog of unresolved security investigations.
- Key risk indicator (KRI): Measures the company’s level of risk, and how its risk profile changes over time. An example for security operations is to use metrics that measure the severity of threats and vulnerabilities being reported by sensors. Another example is to look for places in the security defensive chain events that are happening (e.g. end-point-based events are more “risky” than firewall or WAF events). Finally, make sure you have a good understanding as to business-role the assets involved play. Security events that occur on critical assets present more risk than those on noncritical ones.
- Key control indicator (KCI): Indicates how much control a company has over its environment and its level of risk, or how effectively a particular control is working. Putting this in context with IT security operations, a question to ask is whether you have the necessary controls across all areas of the business – for example, the NIST Cyber Security Framework functional areas (identify, protect, detect, respond and recover). Knowing that these functions have sufficient coverage throughout your defense in depth (devices, applications, networks, data and users) gives you a degree of confidence in your controls.
How to Use These Metrics
The interplay between the performance, risk and control metrics is the key feedback that an organization needs in order to be confident that investments in cyber security are appropriate. Now that we have defined the appropriate use for the individual metrics, let’s see some examples of how to apply them:
- Risk is the probability of bad things happening applied to the business cost of it happening. You can calculate an estimate of the probability by looking at the number, place (where in the defense in depth model) and severity of events measured by sensors. For the impact, or real cost, look at which hosts are involved. Are they where the crown jewels are kept, or more of an extra store-room full of old furniture? Faced with so much data, organizations can be afflicted with “analysis paralysis,” so simplify these measures into risk metrics everyone can understand.
- Performance metrics are meant to show how efficient an organization is at accomplishing its mission. In cyber security, the mission happens to be risk mitigation. So performance is how well you manage your backlog of open security cases, time to resolution, etc. with respect to the staff and systems you have. There are significant parallels to customer support metrics in this category.
- Controls mitigate risks and enable performance. In cyber security, technical (security sensors) and process controls are your bread and butter. They also generate the data that drive risk metrics and allow you to optimize performance. Compliance measures are your friend here. Measure your degree of coverage against a framework such as NIST CSF.
Generating the metrics here seems like a daunting task at first. But, once you start simplifying and categorizing the measures, you will find that you can come to a reasonable set quickly. Then you need to automate their calculation. With experience, you will learn whether you’ve chosen the right KPIs and KRIs, and you can make adjustments as necessary. Getting started can be a challenge for MSSPs, but it’s 80 percent of the battle.
The most important thing to remember is that the statistics coming out of your cyber security systems are not KPIs, KRIs or KCIs. They are just data. Decide what risk performance, risk or control measures you need in order to clearly explain metrics of security operations to the business you support.
Test these on business managers to make sure they resonate, adjust and go again. The more consistent and transparent your measures, the more confidence your clients will have in their security investments.
Putting KPIs, KRIs and KCIs into Practice
On one hand, you have a large amount of security data – the proverbial big data problem. On the other hand, you need actionable output – a list of what to do now to transform your clients’ security programs into a high performance business driver. Metrics will guide your path to success, but generating consistent and reliable information security metrics is hard. So here are a few steps to get you started.
Step 1: Understand your Coverage, Operations, and Compliance Challenges
Security operations involves a set of functions being performed across a set of assets. The NIST Cyber Security Framework (CSF) provides a core list of the functions and the Cyber Defense Matrix from OWASP does a fine job of aligning those functions against a representative set of assets. Categorizing the deployed security products or processes in your client’s environment within the matrix will establish coverage and identify gaps in the program’s architecture.
Operationalizing the matrix by collecting, identifying and assigning the output data from your security products to each cell in the matrix shows evidence of operations and serves as your first step in addressing the ‘big security data’ problem. Gaps between what you thought you had deployed and what actually shows up as evidence of operations will provide you with an immediate ‘to-do list’.
Applying a control framework (such as CIS Top 20, GDPR, or FFIEC) adds depth to each of the intersections by mapping specific security controls to both deployed security products and your client’s assets. The resultant overlay identifies gaps in your compliance effort and your second ‘to-do list’. When combined with your operational to-dos, the entire list can be mapped to a 30, 60, 90-day plan of action with key milestones. Wash, rinse and repeat for each of your lines of business or departments, and you now have a path for your journey.
Step 2: Measure your Efficacy
With security products and processes deployed and more on the way as you move down your path, it is time to measure the effectiveness of each action and ensure its alignment with the business. Recall that operationalizing the matrix served as the first step in solving the big data challenge by categorizing the data and applying business context through the assets in the matrix and each line of business or department.
Enriched with this context, the security data can now be normalized and analyzed to produce key metrics, or as we called them earlier, KPIs, KRIs and KCIs. Examples include the speed of new threats or vulnerabilities for KRIs, the treatment of symptoms or root causes for KPIs, or the reduction defensive workload for KCIs.
With metrics in place, each to-do on your journey can be seen as a resultant change in one or more metrics. What’s more, the value of fixing operational to-dos or implementing a specific control can be measured and communicated specific to the business context it affects. At each milestone on the journey, thresholds for metrics can be set to determine success or identify needed adjustments in the plan.
It’s all about the journey. A successful information security program is not an end-state, but a continually monitored and adjusted compilation of people, process and technologies. Mapping the program’s functions with your client’s assets and required controls provides you the steps needed to mature your program while metrics will keep you honest about how well the program is performing.