Vulnerability Remediation - Do Not Forget Regression Testing

INTRODUCTION

When talking about the remediation activities associated with Vulnerabilities, it has become obvious in a number of instances, that whilst there are well meaning efforts being given to promote vulnerability fixes into Live as fast as possible, that Regression is often introduced in the haste, as an unwanted side effect.

Get the testing of the vulnerability remediation process wrong and you could end up with Customer complaints, unhappy stakeholders, and a lot of unnecessary pain. Literally no one will thank you for having remediated the Vulnerability quickly whilst introducing several potentially even more impactful issues.  

Regression testing is the process of re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change.

TEST, TEST, TEST.

There is often an approach taken when dealing with Vulnerabilities that because they are security related, they are somehow different from other issues/defects and that we need to ‘just fix it and get it in’. This is understandable to some extent, as for people outside the security world, the very words security issue can strike fear and panic!

The view needs to be changed to ‘fix it, test it properly and then get it in’. In other words, lets treat a Vulnerability, at least from a testing perspective, in a similar way to other functional and non-functional defects.

So, for each vulnerability there needs to be two aspects to the testing:

1)     Have we properly remediated the Vulnerability and tested it can either be downgraded or closed?

2)     Have we understood the fix that has been applied, impacted the changes and Regression tested appropriately? (This is the part which tends to get missed)

WE HAVE A VULNERABILITY REPORT – NOW WHAT ?

So, you will have received a Vulnerability report, probably from a Security consultancy or an internal Vulnerability Scanning Engine. This report will contain a number of different vulnerabilities each scored with a corresponding Common Vulnerability Scoring System (CVSS) score.

Fixes to Vulnerabilities come in many different forms, from simple configuration changes, to complex code rewrites and underlying product version changes. An important aspect when thinking about testing of the fixes to the Vulnerabilities, additional to the specific issue itself, is to understand the scale of changes required to enable the fix and to then impact and plan the execution of a test set which exercises that change appropriately.

To remediate the Vulnerabilities you have been passed, someone needs to:

• Prioritise the vulnerabilities into an agreed order. (also known as triage)
• Understand the fixes required and impact assess the changes.
• Engage with the people /teams we need to do the actual fixing for us.
• Engage with the people who will test that the fixes have been successful.
• Engage with the people who will test that other parts of the applications have not been regressed as an unintended consequence of these fixes.

All of these actions align with what would typically happen in any of the other test phases in play, so we should try and think of the process in similar terms to other issues we fix and test as part of standard processes for maintaining and testing applications. By adhering to a test process we can start to get alignment with what we likely already have in place for testing efforts elsewhere in the company, in areas such as Integration Testing, UAT, OAT , Performance Testing and so on.

PRIORITISATION

First thing we need to do is prioritise the vulnerabilities in the order which we want to resolve them. This ordering is often done by descending order by CVSS score. However, this ordering is somewhat flawed as, “out-of-the-box” vulnerabilities are not given business contextualisation. Ideally, we should use a Vulnerability Triage Group (VTG) to apply business context to findings and ascertain fix prioritisation, but that is a topic for another blog post .

Without a VTG, this prioritisation typically takes place best when the security consultancy producing the report are in discussion with you as the person responsible for the remediation and/or wider business cyber security, but also with the network or application teams who can give more context to the vulnerability during testing.

For example, there may be a high vulnerability, which is one of five high vulnerabilities in the report, but one which applies to a server holding the business-critical data. Now that we understand the criticality of the server upon which the vulnerability resides, this issue may become the natural candidate for priority remediation. This process of understanding the vulnerabilities within the organisation context is a good way to order our efforts and allows us to start getting the remediation plan together, with associated timelines for completion ( you can bet you will get asked for this timeline a few times! ).

FIXING AND TESTING.

OK let us assume we now have our agreed priority order which we are going to use to fix the vulnerabilities. Next step you should try and identify the people, the teams, the process, and the departments who are going to be impacted by these remediations.

Let us apply a working example to demonstrate some of the things to be considered.

Let us assume an organisation has a web application which has been developed by a third party. This third party develops the code and delivers into the receiving organisation as and when requested.

There are several things to think about when engaging them. First up we need to establish whether there is an SLA in place for them to resolve security vulnerabilities. There may also be a commercial discussion which is required, as they may have introduced the Vulnerability themselves because of bad code practices. Suffice to say there are many elements to that supplier management which come into play, but for our purposes when considering Regression, we need to establish what level of functional testing they will perform after they have resolved the Vulnerability.

They may not have any agreement to do any testing, so it is well worth finding out at this stage such that you can plan to enhance the level of testing you perform after they have delivered the code back to you.

Unfortunately, in a lot of organisations, the people dealing with all thing’s security related often have little or nothing to do with the teams who perform the other forms of testing against applications. Especially in larger organisations there are likely to be teams in place whose responsibility it is to test applications from both a functional perspective, and from a non-functional, performance and resilience perspective. We need to use the expertise and test automation capabilities of these teams to deliver our remediation activities quickly and safely into the Live estate and avoid promoting code which may well fix the actual Security issue, but break large swathes of the application in doing so.

This is what we are trying to avoid. Do not throw a change into Live because “it’s a security issue”, without testing what else may have been impacted by the fix.

REGRESSION TEST

The key to being able to fix and test quickly, is to have an automated test set, you can rely on to give you significant coverage and speedy execution times. We are no longer living in the days where Regression testing is manually performed and can take an inordinate amount of time. We should have automated test Regression packs, ready to be executed to deal with the scenario our example gives.

We have taken some code; we have tested the Vulnerability has been remediated. All good so far! We however have no idea what damage might have been done in the attempts at resolving that defect as our supplier may not have done any level of Regression testing.

If we are mature enough to have a DevSecOps approach where security is automated into our change process and we have tools like Jenkins orchestrating functional and non-functional automation , then great, we have a mature test process which includes Security and we have likely mitigated the risk somewhat. (Also check what your suppliers are doing in this regard, do they do anything at all ?) If, however, like a lot of organisations, this is not the case, then we need to look to our automated functional test capability to give us assurance that the application functions as we expect. We need to engage the Test Automation teams and ask them to run their Regression packs, either over the whole application or more likely focused on the area of change.

PROVING WE TESTED IT BEFORE WE PROMOTE.

At the point when we have tested both the specific fix and ran a level of Regression over the application(s), we need to look to an Exit from the test phase.

Exits from test serve several purposes, one of which is a particularly useful one for the person responsible for Vulnerability Remediation… probably you. This purpose is that it allows us to document clearly what testing we did and importantly did not do. ( this is particularly important as it is nigh on impossible to test everything, and it allows a good response to the uninformed questions which always seems to be asked..”Did you test everything”. No is the answer and add to that a test exit which documents this, and you have an artefact to produce should it be required.

Some simple Exit criteria might be:

• Were all our planned tests executed?
• What was the test pass rate and is this acceptable?
• Have any issues been raised in the relevant Defect tool and assigned for resolving?
• Were any issues introduced as part of this build and found in Regression test, can they be carried forward into a further release, or do they need to resolved as part of this build?
• Has the actual Security issue been resolved?

Hopefully, this is mandated in your organisation anyway, as a required input to a Change board.

SUMMARY

In summary, try not to be pressured into promoting a change purely because it is a Security related issue, without fully considering the impact of the change and the testing required.

There will of course be instances when risk being carried by the Vulnerability determines that we must promote fixes almost immediately, but there will likely be many more occasions when risk being carried should be balanced against the need to Regression test following fixes being applied.

Having a strong test automation capability which gives an extensive regression test coverage allows us to take and test changes quickly. It means even when the pressure is on to ’get it in’, we can still be safe in the knowledge we will not take the product backwards in our haste to be secure.

Precursor Security assist clients with both Continuous Security Testing and with Automated Functional Test designs and implementations. If we can help further with any guidance or questions you may have please contact us for details.