The Beginners Guide to Vulnerability Triage

This blog has been updated to reflect some of the readily available changes in vulnerability triage over the past few months. This aims to provide an actionable way for organisations to triage and prioritise their fixes without the requirement for more time and resources.

At Precursor, we already do a lot of what’s mentioned here for our clients as part of our normal service offering. By doing so, we’ve been able to reduce our clients average Time-To-Remediate (TTR) to 3.8 days following the discovery of a vulnerability by one of our testers, while ensuring remediation efforts are focused on real-world risk. This blog will hopefully help you see the same results.

What We Are Currently Doing Isn’t Working

Times have changed and the simple fact is that if what we’ve been doing all this time worked, we wouldn’t be seeing countless organisations spending hundreds of thousands in security programs being compromised every other week.

Statistically, on average, organisations will only manage to fix 10% of vulnerabilities. Yes, 10%. When we combine this with the fact that vulnerability exploitation is most likely to occur within the first month following the release of a patch, the issues become obvious.

What we can’t do is magic up a 10-fold increase in resource or make days 240 hours long. What we can do is apply some form of intelligence, logic, and good old common sense to make sure the 10% we fix are impactful and have a meaningful reduction in real-world risk.

Lets Get Started

As with most topics in cyber security we could go on to write multiple books on vulnerability triage. However, for the purpose of keeping this short and sweet we are going to make some assumptions about your current security processes.    

The main assumption we will make is that you perform some level of active security testing. This can range from full penetration tests and prolonged adversary emulation exercises to monthly vulnerability scans with one of the many vulnerability scanning products and solutions out there. It doesn’t really matter which, so long as you’re actively making some effort to identify security vulnerabilities across assets you control.

Whether it’s hardware, software, process or people, the activity of identifying vulnerabilities is ultimately going to leave us with a list of issues that we need to address. Which is why we need to look at the whole Vulnerability Triage and Remediation process.

Traditionally we use CVSS

Traditionally, Penetration Testers will use a combination of the Common Vulnerability Scoring System (CVSS) and some form of impact & probability / risk metric to grade the severity of any identified vulnerabilities based upon the information they have available to them. Combined with the testers experience, report items will then be sub-ordered to ultimately provide a top-down technical report itemising the most severe to least severe vulnerabilities.

This still doesn’t really work

While at first glance it appears that a lot of the hard work is done for us there will be many, many times in your cyber security journey, that will result in a High & Medium impact vulnerabilities being discovered that at first glance, all appear as critical as each other and need to be addressed as a matter of urgency, however when investigated are theoretical, have no public exploit code, have no available information and have never been used by a threat actor in a real world attack.

The result is that our 10% of fixes are directed at vulnerabilities that realistically will not be used against us and vulnerabilities that are genuinely posing a risk to the organisation remain unfixed. Obviously, this is bad.

What we need here, is a methodology that allows us to triage and fix vulnerabilities quickly, and effectively in line with realistic risk to ensure that our efforts are not wasted and make the most impact.

We need a process

For someone in a position of responsibility early in the cyber security process this can be a daunting task, particularly when individuals further up the chain of command are demanding immediate results.  

Step 1. Take a breath and make a plan

The first thing we need to do before any level of active remediation effort takes place, is to come up with an effective remediation plan to ensure a structured approach is followed when deploying changes, as a result of security issues. Without these, one of two things are likely to happen:

1) “The task is too daunting; I have uptime and availability to worry about and this is not causing us an immediate operational impact. I will deal with this when I have time” – Spoiler Alert… The to do list never gets any smaller. The report will likely sit in a file share or a desk drawer until the next test comes around, a customer asks for evidence of security tests or, should the planets align you actually end up with the time to look into this.

2) “Let’s just start at the top and work down”. While this is infinitely better than the first outcome, due to the fact we’re actually trying to address issues, this can still present problems. Problems can arise particularly where remediation efforts are allocated as a secondary task to normal day-to-day function. While we in no way want to disrupt the smooth operating of the organisation, if the task is not given the proper attention. We are likely going to start with the best intentions, but as other tasks get in the way the efforts tail off. The end result of this is that we remediate some or all of the high criticality vulnerabilities but don’t consistently remediate all vulnerabilities.  

If you fall into either of these categories, it’s likely that you have two core problems. The first is that your vulnerability management process lacks enough allocated responsibility and/or resources. The second is that there is no process to analyse and triage the output of vulnerability identification exercises, so that we can produce an actionable plan that allows us to best align our efforts and yield the highest output in the shortest time. So, let’s dive right in…

Your Responsibilities

While not the primary objective of this blog post, it’s still worth touching on the topic of responsibility. Responsibility, or more accurately the lack of it, is a common issue that particularly plagues small and medium enterprises. Just by reading this post, you are already demonstrating that you understand you have some level of responsibility for the protection of your customers’ data. But with a million other things to do and budget dictating that hiring dedicated resource is unfeasible, you are going to have to work with your existing teams to be sure that these vulnerabilities get fixed.

It’s worth remembering that, while the responsibility for deploying fixes at a tactical level can be delegated to relevant teams, the overall responsibility for the protection of data still lies with you. Make sure you track your team’s efforts and allocate timelines that are both achievable and ensure remediation is completed before something bad happens. Once fixes get tested and deployed, ensure that efforts are made to retest so that we can ensure fixes are effective and deployed correctly.

Too many times changes are made without proper understanding of the issue, or worse, with a slapdash “just get it done” approach. The result of either of these is often the continued existence of the security vulnerability under the false understanding that the issue has been fixed. This can all be managed with a proper “Test & Deploy” process, but that’s for another post.

Step 2. Set up a Triage team

However, before we allocate tasks to the relevant teams, we must perform our triage process. For this we need to delegate a separate team whose core function within the vulnerability management process is to rapidly analyse and prioritise incoming vulnerability data. This team will be known as the ‘Vulnerability Triage Group’ (VTG) and should hold between its members experience in the organisation’s IT management, general cyber security risk and wider business risk.

The bottom line is that while you are responsible for the overall protection of your commercial, staff and customers data, you will likely need to work with several teams to implement a complete vulnerability management process. Prioritisation is the responsibility of a core group (the VTG) who have enough significant experience to understand the issues and their impact within your business context.

Triage Guidance

As always, the NCSC have outlined vulnerability management steps here, and it is advised material for anyone who wants to understand the vulnerability management process. As such we’ll lean on its core principles below.

At this point we’ve hopefully allocated responsibility to asset owners for the technical remediation of both current and future vulnerabilities identified across systems under their control. We’ve informed them of this responsibility and the need for this process. We’ve made sure they understand that dedicated time will be allocated to fixing security issues, but they also understand there may be occasions when these fixes take priority, particularly when threat intelligence activities identify malicious actors actively exploiting the vulnerability in the wild or where exploitation is trivial.

We’ve also allocated a core team that hold the correct level of experience and knowledge to understand your vulnerability identification output within the context of your business; the VTG. This group is responsible for the prioritisation of vulnerabilities prior to the delivery to asset owners.

With our team now preparing for incoming tickets we can begin to work through our identification process output. As mentioned, there are situations where, despite the best efforts of the security tester, we’re left with a load of vulnerabilities to remediate, many with high and medium criticality findings and we’re not quite sure where to start.

Like any good strategy we want to make the maximum impact in the least number of moves.  

The process of ordering these vulnerabilities by criticality is called “Triage” which can be loosely defined as “The process whereby previously identified vulnerabilities are evaluated, impacted, prioritised and remediation tasks assigned.”

Why ‘Top-Down’ isn’t always best

We’ve already said vulnerabilities are ordered by CVSS. So why invest the time and resources into triage when it’s basically done for us? Surely, we can just get stuck in.  

No.. We can’t.  

Let’s look at an example: AcmeOrg has around 20 members of staff, and a small IT team of one or two people. For the sake of argument an experienced technician and a junior in training. The CEO has been alarmed by the increasing number of cyber-attacks they see on the news and as such has allocated some budget to perform a penetration testing exercise. Having just performed their first penetration testing exercise across all their assets, from both external and internal perspectives, the team gets dumped with 2 very large reports with a load of red and orange all over them. The CEO reads the report and understands that the issues are not a reflection of the current teams’ efforts, but rather the nature of security coupled with a resource issue… After all, their company is growing and so should their IT team to support them. They tell you that you can sit down with them, go through the Cyber Security Skills Matrix and create a job posting to fill a more permanent position. …………This could not be going any smoother… what’s the catch?  

“In the meantime, we need you to sort all of this report out ASAP. Go get me a timeline”  

Whelp. We still only have a team of two and there’s a lot of vulnerabilities… Where do I start? The report is ordered by CVSS so I’ll just start at the top and work down and come up with the arbitrary figure of 2 weeks because I really don’t know how long it’s going to take and you can get a lot done in 2 weeks.  

The reality is that across the two reports, it’s likely some vulnerabilities with CVSS 9 are going to be less exploitable than some vulnerabilities with a CVSS of say 7.5. For example, that locally exploitable missing Microsoft patch on the machine in the corner that’s sat on its own VLAN with no outbound access is going to be less critical than the known directory traversal on the public web server.  

So, if we take this top down approach the obvious result is that were spending time remediating vulnerabilities that are not likely to be exploited instead of fixing vulnerabilities that are more likely to be exploited for direct impact or used as part of a wider kill chain. Which is not ideal. Taking a little bit of extra time to come up with an effective plan is going to be much more efficient than just spraying update commands across everything we can see.  

Triage Planning

Instead, we could take a morning to assemble our VTG and go through the reports to come up with an effective plan.  

One objective of the VRT is to categorise vulnerabilities under three labels: Fix, Acknowledge, Investigate..

Fix

We’re going to prioritise and deploy fixes for these issues. Regardless of the CVSS, these vulnerabilities pose a risk to our organisation and our ability to maintain Confidentiality, Integrity or Availability (CIA) of our data and systems.  

Acknowledge  

We understand these issues exist, but they aren’t a priority right now, we’ll justify why we aren’t fixing them (In documentation!) and we’ll re-evaluate the issues at a later date.  

Investigate  

We are not sure if we need to fix or acknowledge this vulnerability. There are unknowns that are blocking further action. We’re going to allocate a completion date and investigate in the interim to get more information, so we can move the vulnerability to either the Fix or Acknowledge categories.  

A note on Acknowledgment – It can be tempting at times to just “accept the risk” and start lumping everything into the Acknowledge category. Ultimately this is a business decision, but whenever something ends up in this category you need to document exactly WHY this issue is being acknowledged with sound enough justification. So that if you are compromised and this vulnerability gets exploited you can justify why you didn’t fix it to third parties… remember… by knowing it’s there you’re no longer ignorant and your decisions now could have serious ramifications down the line.

Anyways, we’re making progress, we’ve started to document a vulnerability management process that we can use to align our efforts and make a real impact on our organisation’s security. But how do we decide what vulnerability goes into what box?

Step 3. Prioritise

To prioritise vulnerabilities, it naturally boils down to two questions:  

1) How exploitable is this vulnerability?  

2) If it gets exploited what’s the worst that’s going to happen?  

This line of thinking naturally dictates that externally facing vulnerabilities that will allow an attacker to gain access to data, accounts, systems and anything else we don’t want them to access are going to be a priority. However, ensure you don’t get tunnel vision. On the ground, networks are seldom built with the classic network boundary we all know and love. We’re integrating systems all over the place. From Azure to AWS and Online CRMs the traditional network boundary is becoming blurred. So, make sure you really take time to think about the vulnerability impact and how it will be exploited. This is where the teams wider experience comes into play since the impact of exploitation is at both a technical and business level. For example, we at a tactical level, care an awful lot more about the Remote Code Execution (RCE) vulnerability than the Local file Inclusion (LFI). However, in the wider operating environment we care an awful lot more about the LFI on systems holding our Critical Data that we do about the RCE on a stand-alone printer in the guest network.

Enter EPSS, CISA KEV and Other Intelligence

As we’ve previously mentioned, we need to direct our efforts towards things that have a higher probability of use against our organisation. The problem is that understanding this requires accurate, up to date and actionable real-world intelligence. Thankfully there are some public data sources here that can help us.

CISA Known Exploited Vulnerabilities List (KEV)

The US Government Cybersecurity & Infrastructure Security Agency (CISA) produce a list of vulnerabilities that are known to be exploited by real-world attackers. This list is available here:

• https://www.cisa.gov/known-exploited-vulnerabilities-catalog

If one of your vulnerabilities is on this list, you need to fix it as a priority because its being used in the wild.

Exploit Probability Scoring System (EPSS)

The Exploit Prediction Scoring System (EPSS) is a data-driven effort for estimating the likelihood (probability) that a software vulnerability will be exploited in the wild.

EPSS will give your vulnerabilities a percentage probability that it will be exploited within the next 30-days if seen by an attacker. The data is refreshed every night and helps prioritize vulnerabilities based on what is really happening in the wild.

More info and access to the data is available here:

• https://www.first.org/epss/

Other Intelligence

As part of our penetration test and Continuous Security Testing reports at Precursor, in addition to the above information we overlay threat intelligence from a number of other sources to ensure that organisations can be more efficient in their vulnerability remediation. This ranges from Public Exploit information to closed data sources. Give us a call to see it in action and learn how we can help you stay secure without breaking the bank.

Common Sense

Good old free-of-charge common sense is another great way to supplement the above information and start triaging vulnerabilities. Some simple questions that can be asked are:

• Does this system hold critical data, or is it connected to systems that do?  
• Is there exploit code available for this vulnerability and, if so, is it public?  
• Is it being exploited actively in the wild? (good intelligence can really help with this whole process)  
• What does successful exploitation give an attacker access too, what systems can they see?  
• What are the chances this has been exploited already?  
• How easy is it to detect exploitation of this vulnerability?  
• If this issue is exploited, does it make any other vulnerabilities exploitable?  
• How old is the vulnerability?  
• How much is it going to financially cost to fix this issue? (Quantified analysis is another subject for another post)  
• How much time needs dedicating to fixing the issue?  
• Who is responsible for the asset?  
• Who is responsible for the fix?  
• Do we need to engage any third parties to address this issue?  
• Are there any temporary mitigations we can put in place pending further investigation?  
• Has this vulnerability been identified previously, and risks accepted? – if so, what is the Risk ID? and importantly has the risk changed?  

On a side note: you can see that one of the questions is “What are the chances this has been exploited already?”. Its again not the focus of this post, but there comes a point where you have to decide if you need to move into an incident response phase and start looking for Indicators of Compromise and adversary activity.  

By answering the above questions, we can start to gain a deeper understanding of the vulnerabilities that exist across our assets and as such start to come up with an actionable plan that allows us to make the biggest impact in the least amount of time. Understandably though, these questions require a level of both business and technical input, which in many situations we may not possess. Don’t be afraid to lean on your preferred security consultancy for help in understanding the issues. But, if you’re really stuck, flustered and don’t have the time to perform a full vulnerability triage exercise the following guidelines from the NCSC are a good place to start:  

Priority 1: Fix Internet services and off-the-shelf web applications that can be exploited automatically across the Internet with no user (or attacker) interaction.  

Priority 2: Fix bespoke/niche web applications that can be exploited across the Internet with no user (or attacker) interaction.  

Priority 3: Fix Issues that can be exploited across the Internet with minimal user interaction (workstation vulnerabilities, drive-by downloads, email-based attacks).

Priority 4: Fix issues that can be exploited across the Internet with social engineering of users (malicious applications downloaded from the web or sent via email). These attacks require your users to play a part.  

Compensating Controls

It’s a fact of life (for the majority of us at least) that we can’t afford everything we want, sometimes closing a security vulnerability in the most ideal way is going to cost far too much. It’s also a fact of life that phishers are going to phish, no matter what you’re told by a security vendor you aren’t going to stop 100% of those emails hitting your users. Finally, it’s also a fact of life that no matter how much we all want to pretend it isn’t true, some systems just can’t be upgraded.  

When you find yourself in any of these positions we can’t just say “well we can’t fix it so why bother?”; we need to make the best of a bad situation. We can’t implement the ideal fix, so we need to introduce compensating controls. These are changes we can make to reduce and contain the overall risk posed by the vulnerability to an acceptable level.  For example, an aged legacy system that can’t be patched might be better protected by a dedicated firewall and a segmented network.

Be under no illusion compensating controls can often be more difficult to implement than the ideal fix. They are in no way the easy and cheap option and should only be used when there is an absolute blocker on the preferred remediation.  

Final Words

We’ve shown that a structured approach to vulnerability triage is an essential part of an efficient vulnerability management programme and, while it can take some time and effort it’s well worth it.  

If you are looking for any guidance on how to best implement Vulnerability Triage across your organisation please get in touch at info@precursorsecurity.com and we’d be happy to share any guidance where we can.