Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?

Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.

My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.

Fun little hacking for posturing is one thing. But now that money is involved, we see just how unethical this hacking really is. Manuel Kalloff has a great summary of the true nature of these impacts here: https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/

Note the massive uptick in incidents. These people aren’t doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won’t end there so you really need to pay attention!

You can also track this event here: https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/

(please note – these are all external links and I can’t validate any content in them because of that)

I will try to keep updates to this page as much as I am able – but for now please realize that until the dates below every NetScaler ADC is vulnerable without the code… and possibly even then.

Update 1.20.2020

Citrix has increased their efforts on this remediation with new firmware. Initial releases are out now, with key releases like 12.1 following on Jan 24th. Blog with details here: https://www.citrix.com/blogs/2020/01/19/vulnerability-update-first-permanent-fixes-available-timeline-accelerated/

Update 1.18.2020

I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours’ for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.

What we know about CVE-2019-19781

The vulnerability affects all supported product versions and all supported platforms:

• Citrix ADC and Citrix Gateway version 13.0 all supported builds

• Citrix ADC and NetScaler Gateway version 12.1 all supported builds

• Citrix ADC and NetScaler Gateway version 12.0 all supported builds

• Citrix ADC and NetScaler Gateway version 11.1 all supported builds

• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds

Update Jan 17 2020 – ” This vulnerability also affects certain deployments of two older versions of our Citrix SD-WAN WANOP product versions 10.2.6 and version 11.0.3 ” from https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

What this Citrix ADC Vulnerability Impacts

Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.

Update 1.20.2020 – I’d like to add some additional POSSIBLE impacts, including the possibility that ALL accounts that have gone thru the Citrix Gateway should have their passwords changed right away in addition to other potential impacts. Please review Thomas’s article for additional information.

One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you’ll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks.
No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don’t be that person to use hurtful hashtags or spread fear rather than solutions.

Action Steps for CVE-2019-19781

Last update – 1.17.2020

Fix Script for Citrix NetScaler ADC

Use the instructions at https://support.citrix.com/article/CTX267679 right now. The extended Citrix community is working on additional scripts.
The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down’ from the other, you need to replace it with one from your keyboard.
Fortunately – the fix is quick but does require a reboot to take full effect.

UPDATE 1.17.2020

Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I’d do so now.


Check for Vulnerability in your ADC

The US Government has released a method of checking for this flaw. See https://www.us-cert.gov/ncas/current-activity/2020/01/13/cisa-releases-test-citrix-adc-and-gateway-vulnerability and https://github.com/cisagov/check-cve-2019-19781

1.17.2020 – Citrix also released this at https://support.citrix.com/article/CTX269180

Update – 1-14-2020

If someone is cryptomining on your ADC- you’ll see high utilization – but there’s a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.

Here’s what do to. After you’ve run the prevention script and rebooted, get into the shell, or just enter

shell top -n 10

What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON’T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you’re satisfied Ctrl-C will take you out of that.

However, in my mind, cryptomining is a secondary concern. Your company’s information may have been exposed at some levels that have not yet been fully determined.

The big indicator of a compromise at this point is .xml files in directories they don’t belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you’ve been exploited, exporting your configuration and configuring from fresh firmware isn’t a bad idea.

If you are compromised:

  1. Take the ADC off the network.
  2. Change the password of any LDAP or other AD/network accounts stored on the ADC.
  3. Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
  4. If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
    1. Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
    2. Disconnect the network before starting
    3. Start the machine and verify using the console that the VPX does not appear compromised
    4. Change the nsroot password
    5. Attach the internal network only
    6. Run the fix (alternatively- type this via the console to be safer)
    7. attach the external network
    8. Keep an eye on the logs
  5. Replace SSL Certificates on the appliance at your earliest opportunity

Timeline and Updates

Jan 13 2020
Citrix has announced a timeline for ADC firmware that will include fixes.

Note- these are the initial timelines superseded on Jan 19th

VersionRefresh BuildExpected Release Date
10.510.5.70.x31st January 2020 January 2020 January 2020 January 2020 January 2020

Jan 14 2020

I have started working with clients to remediate compromises and double-check other clients. I’ve updated some suggested quick things above.

I’m tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!

…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.

Jan 16 2020

Citrix released an officially supported way to scan for the vulnerability, though it is not exactly user friendly https://support.citrix.com/article/CTX269180

Jan 17 2020

Citrix Posted this blog article clarifying a few things: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

DJ Eshelman announced that he will be working with a few other industry professionals to create a video and community support course offering to help people remediate this issue.

Jan 19 2020

Citrix blog post with updated firmware releases.

Citrix ADC and Citrix Gateway
VersionRefresh BuildRelease Date 19, 2020 19, 2020 24, 2020
10.510.5.70.xJanuary 24, 2020 24, 2020
ReleaseCitrix ADC ReleaseRelease Date
10.2.611.1.51.615January 24, 2020
11.0.311.1.51.615January 24, 2020
I’m Writing a Book on IT Project & Services Methodology and I’d Love Your Help

I’m Writing a Book on IT Project & Services Methodology and I’d Love Your Help

I have learned a few things over the course of the last 20 years. These are things I’ve been teaching to other consultants, teams and coaching clients for years, so I’ve decided it was time to put them into a book!

The book focuses on what we call a Methodology: a repeatable process for working that is predictable enough for others to recognize and learn. Rather than a complex process that requires a certifiation to even understand, I have found that successful organizations use a methodology that can be understood by everyone from sales, services, support and up to the C-Suite.

In other words- if you’re looking for another book on Agile or other DevOps methods… while some of those can be implemented inside of this – I have found that these methods tend to isolate IT and isn’t always effective. Why? Because if your entire team isn’t on-board, it tends to fall apart. So what happens when someone new comes in? Or a key person leaves? Or… the team gets lax in taking actions? What happens when moving too fast causes outages from risks that weren’t properly identified? I call it a Resume Generating Opportunity. And it is exactly what I want to see people avoid.

That’s why I have kept this book outside of the theoretical and describe EXACTLY what works NOW, and has been easy to understand everywhere I have taken it. The book is helpful if you are in sales, a service desk or a seasoned consultant. Everyone has something they can learn and I have plenty to teach!


I need your help! This survey will help me determine the best title and keywords to use to make sure everyone sees this book who needs it! I’d very much appreciate your vote!

EUC State of the Union 2019

There is a lot of marketing, hype and FUD (Fear, Uncertainty & Doubt) thrown around in the EUC (End User Computing) space. So I’m always grateful for true survey results to see the true trends in the industry. So, let’s have a look!

You can register for and download the report here: https://vdilikeapro.com/announcing-the-state-of-euc-2019/

My friends Mark Plettenberg, Ruben Spruijt and Christiaan Brinkhoff have compiled this 73-page report that speaks the truth about the trends in VDI and EUC in general. Worldwide, just under 600 people turned in surveys which the team decoded and compared to previous years. Here are some of my personal favorite highlights.

Highlights of the 2019 State of the Union

  • Healthcare continues to be the largest business vertical staying on-premises for EUC.
  • Age of environments is interesting – VDI environments are often exceeding 5 years without significant updates. New designs had been declining but saw an uptick this year.
  • Citrix still dominates the VDI market but decreased by 7% to just over 50% of survey responses this year.
  • Nearly 22% are still using Windows 7
  • FSLogix use increased by about 2% – note that this was probably before people really knew they can get it for free now… But Citrix UPM is still the most popular choice.
  • PVS is still popular for imaging though usage is declining. MCS is almost neck-and-neck with PVS.
  • 39% of respondents were from North America with 50 overall countries represented.
  • The majority of companies have between 1,000 and 4,999 users
  • An overwhelming percentage (76%) are still using On-Premises Server Based Computing (RDSH or Multi-Session OS) – this only decreased by 2%.
  • VMware vSphere is still over 50% of the market, however, the big note I took from it was that Citrix, Microsoft and Nutanix Hypervisors are all increasing while VMware is decreasing. KVM (driven mostly by Nutanix) is up significantly this year.
  • More and more people continue to think of VDI as stateless or non-persistent.
  • 5.6% of respondents say they use Citrix EdgeSight for monitoring. If that doesn’t shock you, it’s overall position on the chart will. Not bad for being dead for nearly a decade.

I could go on- but honestly, I think you should download the guide and compare your results!

Configure Your NetScaler (Citrix ADC) 13 with Carl Stalhood’s new guides

Configure Your NetScaler (Citrix ADC) 13 with Carl Stalhood’s new guides

Carl Stalhood is at it again, with new build guides for Citrix ADC (formerly NetScaler). He has been so helpful over the years so I thought I’d boost the signal to his site a bit. He is a big reason I don’t currently make my own guides. Why would I need to? His are great!
We will keep this post up to date as best we can – Carl moves faster than we do, though!



By With a Little Help from Our Friends

ByteSized Book logo