Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?

Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.

My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.

Fun little hacking for posturing is one thing. But now that money is involved, we see just how unethical this hacking really is. Manuel Kalloff has a great summary of the true nature of these impacts here: https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/

Note the massive uptick in incidents. These people aren’t doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won’t end there so you really need to pay attention!

You can also track this event here: https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/

(please note – these are all external links and I can’t validate any content in them because of that)

I will try to keep updates to this page as much as I am able – but for now please realize that until the dates below every NetScaler ADC is vulnerable without the code… and possibly even then.

Update 1.20.2020

Citrix has increased their efforts on this remediation with new firmware. Initial releases are out now, with key releases like 12.1 following on Jan 24th. Blog with details here: https://www.citrix.com/blogs/2020/01/19/vulnerability-update-first-permanent-fixes-available-timeline-accelerated/

Update 1.18.2020

I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours’ for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.

What we know about CVE-2019-19781

The vulnerability affects all supported product versions and all supported platforms:

• Citrix ADC and Citrix Gateway version 13.0 all supported builds

• Citrix ADC and NetScaler Gateway version 12.1 all supported builds

• Citrix ADC and NetScaler Gateway version 12.0 all supported builds

• Citrix ADC and NetScaler Gateway version 11.1 all supported builds

• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds

Update Jan 17 2020 – ” This vulnerability also affects certain deployments of two older versions of our Citrix SD-WAN WANOP product versions 10.2.6 and version 11.0.3 ” from https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

What this Citrix ADC Vulnerability Impacts

Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.

Update 1.20.2020 – I’d like to add some additional POSSIBLE impacts, including the possibility that ALL accounts that have gone thru the Citrix Gateway should have their passwords changed right away in addition to other potential impacts. Please review Thomas’s article for additional information.

One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you’ll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks.
No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don’t be that person to use hurtful hashtags or spread fear rather than solutions.

Action Steps for CVE-2019-19781

Last update – 1.17.2020

Fix Script for Citrix NetScaler ADC

Use the instructions at https://support.citrix.com/article/CTX267679 right now. The extended Citrix community is working on additional scripts.
The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down’ from the other, you need to replace it with one from your keyboard.
Fortunately – the fix is quick but does require a reboot to take full effect.

UPDATE 1.17.2020

Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I’d do so now.


Check for Vulnerability in your ADC

The US Government has released a method of checking for this flaw. See https://www.us-cert.gov/ncas/current-activity/2020/01/13/cisa-releases-test-citrix-adc-and-gateway-vulnerability and https://github.com/cisagov/check-cve-2019-19781

1.17.2020 – Citrix also released this at https://support.citrix.com/article/CTX269180

Update – 1-14-2020

If someone is cryptomining on your ADC- you’ll see high utilization – but there’s a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.

Here’s what do to. After you’ve run the prevention script and rebooted, get into the shell, or just enter

shell top -n 10

What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON’T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you’re satisfied Ctrl-C will take you out of that.

However, in my mind, cryptomining is a secondary concern. Your company’s information may have been exposed at some levels that have not yet been fully determined.

The big indicator of a compromise at this point is .xml files in directories they don’t belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you’ve been exploited, exporting your configuration and configuring from fresh firmware isn’t a bad idea.

If you are compromised:

  1. Take the ADC off the network.
  2. Change the password of any LDAP or other AD/network accounts stored on the ADC.
  3. Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
  4. If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
    1. Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
    2. Disconnect the network before starting
    3. Start the machine and verify using the console that the VPX does not appear compromised
    4. Change the nsroot password
    5. Attach the internal network only
    6. Run the fix (alternatively- type this via the console to be safer)
    7. attach the external network
    8. Keep an eye on the logs
  5. Replace SSL Certificates on the appliance at your earliest opportunity

Timeline and Updates

Jan 13 2020
Citrix has announced a timeline for ADC firmware that will include fixes.

Note- these are the initial timelines superseded on Jan 19th

VersionRefresh BuildExpected Release Date
10.510.5.70.x31st January 2020 January 2020 January 2020 January 2020 January 2020

Jan 14 2020

I have started working with clients to remediate compromises and double-check other clients. I’ve updated some suggested quick things above.

I’m tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!

…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.

Jan 16 2020

Citrix released an officially supported way to scan for the vulnerability, though it is not exactly user friendly https://support.citrix.com/article/CTX269180

Jan 17 2020

Citrix Posted this blog article clarifying a few things: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

DJ Eshelman announced that he will be working with a few other industry professionals to create a video and community support course offering to help people remediate this issue.

Jan 19 2020

Citrix blog post with updated firmware releases.

Citrix ADC and Citrix Gateway
VersionRefresh BuildRelease Date 19, 2020 19, 2020 24, 2020
10.510.5.70.xJanuary 24, 2020 24, 2020
ReleaseCitrix ADC ReleaseRelease Date
10.2.611.1.51.615January 24, 2020
11.0.311.1.51.615January 24, 2020
NetScaler Security for the XenApp Dummy – Part 1: Assess

NetScaler Security for the XenApp Dummy – Part 1: Assess

So you have this “NetScaler” thing to front end your XenApp or XenDesktop environment… But maybe you are like me and NetScaler Security isn’t what you spend most of your day dealing with. So, how can you make sure in light of recent security threats that it is running properly? In a post in 2016 I discussed how to get an A+ Rating at SSL Labs for your NetScaler Gateway in under 5 minutes. I figured it was time for an update for 2017 taking some new things into consideration but approach this from the point of view of someone like me that isn’t “A NetScaler Person.”

(last update: January, 2019)


NetScaler Rating at SSL Labs to A+!

NetScaler Rating at SSL Labs to A+!

[*updated 1/4/2019 with new link for Citrix article]

Steven Wright of Citrix Consulting has released another guidance for getting an A+ NetScaler Rating at SSL Labs (SSLLabs.com) on June 9th, 2016. The good news is that I’ve validated it works- read on to see the proof!

Why You Want an A+ NetScaler Rating at SSLLabs.com

Security is very much front-of-mind these days, and fortunately SSLlabs.com has a tool to scan your site, including NetScaler Gateway, to detect known problems against current threats.

In case you missed it, you have a whole new reason to re-visit your NetScaler SSL configuration, even if it is a VPX which previously didn’t support nifty security like TLS 1.2. This changed after the last round of updates, so you no longer are forced into an MPX to get good security (though admittedly the CPU usage is a bit higher without the offload chip offered in the MPX and SDX platforms).

If you are running a NetScaler VPX, your out-of-the-box configuration will likely give you a NetScaler Rating of either an F or a C in most cases. Around here, we go for the big grade- so here’s how to get an A+ NetScaler Rating, even with a VPX.

Words of Warning

A few caveats that I know of – First off- I don’t really consider myself an authority on NetScaler, so take all of this with a grain of salt and ALWAYS TEST BEFORE YOU GO LIVE IN PRODUCTION. Messing with SSL ciphers can cause outages, especially for NetScaler Gateway.

Second, if you need to support older clients, especially Windows XP clients, be VERY CAREFUL deploying all of these settings. You may be stuck with a “C” score for needing SSL v3 to stay turned on in some cases. Even a C rating can still be very secure, this is just how SSLLabs.com rates things even if there’s just one attack vector left (unfortunately, SSL is a big one).

But…  if not, you can get a score that looks more like this:

SSLLabs.com A+ Rating

What an A+ Rating looks like from a NetScaler Gateway VPX

Before we go further, I want to reiterate that I’m just validating what someone else created- don’t credit me with this, Steven Wright and Citrix Consulting Services (CCS) did all the work making this possible! Even though I still do occasional work for CCS, I want to make sure noone gets confused!

Previously, you may have looked here: https://www.citrix.com/blogs/2015/05/22/scoring-an-a-at-ssllabs-com-with-citrix-netscaler-the-sequel/  This was a good guide, but consider this article a replacement:https://www.citrix.com/blogs/2016/06/09/scoring-an-a-at-ssllabs-com-with-citrix-netscaler-2016-update/ 

But… in 2018 it was updated again: https://www.citrix.com/blogs/2018/05/16/scoring-an-a-at-ssllabs-com-with-citrix-netscaler-q2-2018-update/

Tested Configuration

The nice thing here is that the blog article has all the steps you need, so break out that puTTY connection and get started!

First things first- note your current rating at SSLlabs.com – I typically do NOT share my results, but feel free if you like to brag.

My configuration included a more modern GoDaddy SSL cert with SHA256 and RSA 2048 strength on a NetScaler VPX 200 with the Enterprise license.

I tested this with firmware 11.0 65.72.nc using the NetScaler Gateway wizard. In my case, it works, so don’t hate me for taking a shortcut 🙂

As I mentioned above- this gave me a NetScaler Rating of “C”. You can test yours by going back to SSLLabs.com and hitting ‘clear cache’ to re-test.

SSLLabs.com NetScaler Rating of C

SSLLabs C Rating on NetScaler VPX

Going from C to B

  • Disable SSL v3
  • I Disabled TLS 1 and 1.1
  • I tried first enabling ECDHE cipher group settings included as a default

Not too bad- a Solid B with this change! I thought it would be an A- but I think there may be a few things in the ECDHE group that will rob you of the rating. You’ll need to define your ciphers manually.

SSLLabs B Rating on NetScaler VPXSSLLabs B Rating on NetScaler VPX

Getting a NetScaler Rating of A+

  • Removed Ciphers (all)
  • Implemented STS (Strict Transport Security)
  • Added the cipher lists that Steven came up with, below
  • Bound the new cipher sets and made sure to use the ECC Curve configuration
Here's the commands to use in the CLI- note that everything in BOLD ITALIC is a name you will need to give it yourself, not a specific command.

add ssl cipher custom-ssllabs-cipher
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1.2-ECDHE-RSA-AES256-GCM-SHA384
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1.2-ECDHE-RSA-AES128-GCM-SHA256
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1.2-ECDHE-RSA-AES-256-SHA384
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1.2-ECDHE-RSA-AES-128-SHA256
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1-ECDHE-RSA-AES256-SHA
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1-ECDHE-RSA-AES128-SHA
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1.2-DHE-RSA-AES256-GCM-SHA384
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1.2-DHE-RSA-AES128-GCM-SHA256
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1-DHE-RSA-AES-256-CBC-SHA
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1-DHE-RSA-AES-128-CBC-SHA
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1-AES-256-CBC-SHA
bind ssl cipher custom-ssllabs-cipher -cipherName TLS1-AES-128-CBC-SHA
bind ssl cipher custom-ssllabs-cipher -cipherName SSL3-DES-CBC3-SHA
unbind ssl vserver DJ_NSG -cipherName DEFAULT
bind ssl vserver DJ_NSG -cipherName custom-ssllabs-cipher
bind ssl vserver DJ_NSG -eccCurveName ALL

  • Next, I needed to allow secure renegotiation, and enable STS on my NetScaler Gateway
set ssl parameter -denySSLReneg FRONTEND_CLIENT
add rewrite action insert_STS_header insert_http_header Strict-Transport-Security "\"max-age=157680000\""
add rewrite policy enforce_STS true insert_STS_header
bind vpn vserver DJ_NSG -policy enforce_STS -priority 100 -gotoPriorityExpression NEXT -type RESPONSE

SSLLabs.com NetScaler Rating A+

That’s it! In under 5 minutes, going from NetScaler Rating at SSLLabs.com of C to an A+!!!

I anticipate this should work on an MPX just as well, if not better- but I haven’t yet tested it. If you have- comment below with the firmware version you used and your score!

Feel free to share this with your friends, and I welcome any feedback below- but make sure if you have anything that definitely needs Steven’s attention to leave a comment at https://www.citrix.com/blogs/2016/06/09/scoring-an-a-at-ssllabs-com-with-citrix-netscaler-2016-update/


By With a Little Help from Our Friends

ByteSized Book logo