Incident Response in Citrix Environments

Introduction

Citrix is a well-adopted and known technology utilised by most organisations. At the time of writing this blog, there are over 60,000 results for Citrix via Shodan and over 10,000 are in the UK. In this blog, we’ll be delving into the threat actors that target Citrix and the vulnerabilities that allow these threats to manifest intrusions into full scale ransomware that we see destroying the economy on a weekly basis.

Our CREST-Accredited team have over 10,000 hours of experience responding to security incidents, with a large chunk of that time spent responding to ransomware incidents where exploitation of Citrix – namely ADC/Netscaler - was the initial access vector. We’ll share stats on the key vulnerabilities impacting Citrix, strategies to protect Citrix and share key commands, scripts and tools to keep in your IT teams arsenal should an incident occur.

The Citrix Vulnerability Landscape

When our Incident Response team “land” on an incident to perform scoping, they gather a plethora of information about the organisation, sometimes from multiple stakeholders. Utilising this information, plausible scenarios can be hypothesised, producing an initial investigation path. Two of the key categories they scope is remote access and virtualisation technologies.When Citrix arises as an answer to either of these categories, it tends to be the vector that allowed the initial access to occur.

This is because multiple ransomware groups have targeted Citrix over the last 24 months, exploiting critical vulnerabilities that allow them to gain remote access, high privileges and code execution. This risk, coupled with Citrix being closely linked to the organisations Active Directory domain and enterprise network is a recipe for ransomware to thrive.

To assess the landscape, there are two key terms we must first understand; CISA Known Exploited Vulnerabilities (CISA KEV) and Exploit Predictability Scoring System (EPSS). KEV is a project maintained by CISA to track vulnerabilities that are known to be exploited in attacks, ransomware included. In light of CVSS being heavily relied on for patching, the industry learned that some critical CVSS vulnerabilities were not worth patching, due to the fact they weren’t being exploited, enter EPSS to solve that problem. EPSS is a project primarily maintained by FIRST.org, of which some of our Precursor staff are liaison members of, continuing their contribution to important projects such as EPSS. EPSS is used to predict the exploitation likelihood of a vulnerability, highlighting that even vulnerabilities with a low CVSS score can cause impact.

Utilising CISA KEV and EPSS, we were able to:

1.        Find all vulnerabilities relating to Citrix that have been known to be exploited in ransomware campaigns.

2.        Calculate the predicted exploitability score of those vulnerabilities, to help with patching prioritisation.

Out of the 16 vulnerabilities in the KEV database for Citrix, 5 of these are known to be exploited to further a ransomware-based objective, we then also added their CVSS score and EPSS rating:

For sake of ease, the EPSS score has been expressed as a percentage out of 100, where 100 would be the highest EPSS likelihood. EPSS provides this score as a likelihood of observing exploitation activity in the next 30 days.

• CVE-2023-3519
  • EPSS: 96%
  • CVSS Base Score: 9.8 (Critical)
• CVE-2023-4966
  • EPSS: 97%
  • CVSS Base Score: 9.4 (Critical)
• CVE-2019-13608
  • EPSS: 0.6%
  • CVSS Base Score: 7.5 (High)
• CVE-2019-19781
  • EPSS:96%
  • CVSS Base Score: 9.8 (Critical)
• CVE-2019-11634
  • EPSS: 24%
  • CVSS Base Score: 9.8 (Critical)

As you can see, at least 3 of those vulnerabilities still have an almost certain level of likelihood for any vulnerable organisations to see exploitation within the next 30 days. We must highlight that this is being written in August 2024, for one of those vulnerabilities, that is almost 5 years on from when it was assigned.

The Threat Actors Behind the Vulnerabilities

During an incident response engagement, one of the next key stages following scoping of the client environment and incident is to perform threat intelligence and build a threat informed profile of what/who is operating in the environment. Doing so allows us to establish a potential pattern of life, motive, tools used and potential indicators of compromise to look for and contain.

Precursor’s curated threat intelligence platform offers upthe following related threat actors, known for exploiting Citrix:

·      LockBit

·      Akira

·      NoEscape

·      BlackBasta

All of which our team have had experience responding to incidents caused by these groups. LockBit in particular were well known for orchestrating an incident against the UKs Royal Mail service, whilst BlackBasta were known for their encryption techniques, including their speed and volume of exfiltration.

‍

How to check if your Citrix has been exploited

We realise that in a pressurised scenario, appointing an IR team or engaging with insurance can be challenging and sometimes time-consuming during an already resource draining scenario. Below, we have provided you with some key commands you can run on Citrix hosts along with references to tools that already exist to respond to such an incident, so that you can scope key answers whilst waiting for expert assistance.

It should be noted that depending on the nature of the vulnerability being exploited, you may or may not have results from these commands, you should always consult experts with purpose-built tooling and Precursor Security accept no liability or damages from utilising these commands.

Find .php files in unexpected locations with suspicious permissions set

find /var/netscaler/logon/ /var/vpn/ /var/netscaler/ns_gui//netscaler/portal/templates /var/tmp/netscaler/portal/templates/netscaler/portal/scripts /vpn/themes /tmp -type f -name "*.php" \\(-perm 0777 -o -perm 0666 -o -perm 0600 -o -perm 0700 \\) -exec ls -l {} +

Check your bash history file for commands typically executed by ransomware actors on Citrix hosts

grep -E "whoami\\$|cat/flash/nsconfig/keys|ldapsearch|chmod \\+x /tmp|openssl des3|ping -c 1|cp/bin/sh|chmod \\+s /var|echo \\<?php" /var/log/bash.log/var/log/notice.log /var/log/sh.log

Check for processes running under the context of ‘nobody’

ps auxw | grep ^nobody | grep -v /bin/httpd | grep -v grep |grep -v "/tests/")

Other tools

Google (Formerly Mandiant) have shared a tool that scansCitrix hosts for specific indicators of compromise pertaining to CVE-2023-3519: mandiant/citrix-ioc-scanner-cve-2023-3519(github.com)

Summary

When responding to complex incidents involving exploitation of remote access and virtualisation infrastructure, you should always consult an expert team such as Precursor Security. Moreover, early indicators of an incident like this can be treated as the tip of the iceberg, as already detailed, exploitation of these vulnerabilities typically leads to a widescale domain compromise where recovery can be lengthy and costly for the businePrecursor Security offers several services to help bolster your organisation against the threats targeting Citrix environments.