Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?

by DJ The Citrix Coach | Jan 13, 2020 | Citrix, NetScaler, NetScaler Gateway

Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.

My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.

Fun little hacking for posturing is one thing. But now that money is involved, we see just how unethical this hacking really is. Manuel Kalloff has a great summary of the true nature of these impacts here: https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/

Note the massive uptick in incidents. These people aren’t doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won’t end there so you really need to pay attention!

You can also track this event here: https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/

(please note – these are all external links and I can’t validate any content in them because of that)

I will try to keep updates to this page as much as I am able – but for now please realize that until the dates below every NetScaler ADC is vulnerable without the code… and possibly even then.

Update 1.20.2020

Citrix has increased their efforts on this remediation with new firmware. Initial releases are out now, with key releases like 12.1 following on Jan 24th. Blog with details here: https://www.citrix.com/blogs/2020/01/19/vulnerability-update-first-permanent-fixes-available-timeline-accelerated/

Update 1.18.2020

I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours’ for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.

What we know about CVE-2019-19781

The vulnerability affects all supported product versions and all supported platforms:

• Citrix ADC and Citrix Gateway version 13.0 all supported builds

• Citrix ADC and NetScaler Gateway version 12.1 all supported builds

• Citrix ADC and NetScaler Gateway version 12.0 all supported builds

• Citrix ADC and NetScaler Gateway version 11.1 all supported builds

• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds

https://support.citrix.com/article/CTX267027
Update Jan 17 2020 – ” This vulnerability also affects certain deployments of two older versions of our Citrix SD-WAN WANOP product versions 10.2.6 and version 11.0.3 ” from https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

What this Citrix ADC Vulnerability Impacts

Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.

One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you’ll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks.
No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don’t be that person to use hurtful hashtags or spread fear rather than solutions.

Action Steps for CVE-2019-19781

Last update – 1.17.2020

Fix Script for Citrix NetScaler ADC

Use the instructions at https://support.citrix.com/article/CTX267679 right now. The extended Citrix community is working on additional scripts.
The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down’ from the other, you need to replace it with one from your keyboard.
Fortunately – the fix is quick but does require a reboot to take full effect.

UPDATE 1.17.2020

Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I’d do so now.

https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

Check for Vulnerability in your ADC

The US Government has released a method of checking for this flaw. See https://www.us-cert.gov/ncas/current-activity/2020/01/13/cisa-releases-test-citrix-adc-and-gateway-vulnerability and https://github.com/cisagov/check-cve-2019-19781

1.17.2020 – Citrix also released this at https://support.citrix.com/article/CTX269180

Update – 1-14-2020

If someone is cryptomining on your ADC- you’ll see high utilization – but there’s a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.

Here’s what do to. After you’ve run the prevention script and rebooted, get into the shell, or just enter

shell top -n 10

What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON’T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you’re satisfied Ctrl-C will take you out of that.

However, in my mind, cryptomining is a secondary concern. Your company’s information may have been exposed at some levels that have not yet been fully determined.

The big indicator of a compromise at this point is .xml files in directories they don’t belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you’ve been exploited, exporting your configuration and configuring from fresh firmware isn’t a bad idea.

If you are compromised:

1. Take the ADC off the network.
2. Change the password of any LDAP or other AD/network accounts stored on the ADC.
3. Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
4. If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
   1. Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
   2. Disconnect the network before starting
   3. Start the machine and verify using the console that the VPX does not appear compromised
   4. Change the nsroot password
   5. Attach the internal network only
   6. Run the fix (alternatively- type this via the console to be safer)
   7. attach the external network
   8. Keep an eye on the logs
5. Replace SSL Certificates on the appliance at your earliest opportunity

Timeline and Updates

Jan 13 2020
Citrix has announced a timeline for ADC firmware that will include fixes.

Note- these are the initial timelines superseded on Jan 19th

Jan 14 2020

I have started working with clients to remediate compromises and double-check other clients. I’ve updated some suggested quick things above.

I’m tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!

…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.

Jan 16 2020

Citrix released an officially supported way to scan for the vulnerability, though it is not exactly user friendly https://support.citrix.com/article/CTX269180

Jan 17 2020

Citrix Posted this blog article clarifying a few things: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

DJ Eshelman announced that he will be working with a few other industry professionals to create a video and community support course offering to help people remediate this issue.

Jan 19 2020

Citrix blog post with updated firmware releases.