First things first – don't panic if you are one of the literally thousands of companies that were affected by CVE-2019-19781… because everyone was! But just because I say don't panic doesn't mean do nothing. I (DJ) have been working with several clients, experts and Citrix around solutions and I think we are in a place where most people have at least stopped the bleeding… but I grow concerned that many more have not.
Every. Single. Citrix ADC (NetScaler) was vulnerable and should be assumed to have been a target.
Some sentences are really hard to write. That last one was definitely one of them. But that all pales in comparison to the implications. The reality is- not only were several thousand companies not ready to handle this threat, my industry friends remain concerned that several thousand customers have not remediated because they didn't know how. Citrix and several sites including my own have given instructions – but if you don't know a few key things about the ADC and how to manage it outside of the GUI (Web Interface) then there is a confidence problem that has kept people from fixing the issue out of fear of locking out their users or making matters worse.
This has kept me busy at home and at hotel rooms for the last few months and I've come to realize I don't have the ability to help everyone. Here's what I can tell you:
Citrix ADC (NetScaler) VPX with a backup before December has been able to be patched and remediation performed in about 3 consulting hours.
Citrix ADC VPX with no backups have typically taken between 4 and 5 consulting hours.
Citrix ADC MPX (hardware platforms) are a particular challenge and in some cases have taken several evenings of downtime to resolve.
The skills needed to properly assess breaches, prevent them and fully remediate from the real threats of CVE-2019-19781 are those that are not commonly used, even by those that administrate Citrix environments.
Many customers deployed Citrix ADCs as a replacement to legacy Secure Gateways and have zero skills administrating NetScaler/ADCs, depending on the contractors that deployed them – contractors that often have moved on or are now so busy with these efforts on top of their existing workloads that it has become impossible to schedule help for many of them.
Worst of all – even though this has been a very popular and well documented case (my article “Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?” is the most popular article I've ever had. It eclipsed a nearly 3 year old popular article in a few weeks) many with Citrix NetScalers don't even know about the issue.
To help, I've developed a kind of a checklist/lesson plan, using my RiskLESS Methodology. Expand each section to see the detail of what needs to be done and the tools and knowledge you will need to be aware of to make it happen.
Understand
Understand the Threat
History of the issue
What is important to know about how the ADC works in this regard
What Hackers are doing (known threats)
What Citrix is Doing
Additional considerations
LDAP threats – high potential for network compromise and backdoors being setup up undetected
If LDAP account was also domain or elevated admin – critical to change password but also begin changing other domain accounts
SSL Key decode threats and need to re-key
NSROOT and any other local account passwords need to be changed
Document your findings of the exposure to management
Collect Passwords and store securely
NSROOT
LDAP account
myCitrix account
SSL provider
Remediation Considerations
Understand the remediation differences between VPX, MPX and SDX
If VPX/SDX – see if there are recoverable backups from before Dec 17th 2019 (if so, celebrate)
HA and Cluster considerations
Note to check all nodes for compromise – may make recovery easier
Firmware version considerations
Some versions of the firmware the remediation script did not work
Disclosure of breach and description to management
If LDAP account was elevated, severe danger requiring rapid disclosure to management and legal
Plan
Fast Remediation Steps
Determine what steps to take immediately and what will wait for a longer change window if this will be required (ie, MPX when access to the gateway will be needed during change window)
Emergency change considerations – brief outage for reboot
Apply remediation script and reboot
Create temp LDAP account password
Change LDAP account and/or password
Written Plan for Extended Remediation
Write out a plan of action based on your platform
Instructions for VPX
clean install
Plan for VPX recovery
Plan for HA secondary clean upgrade + fresh install primary
Instructions for SDX
Instructions for MPX
Special consideration for firmware ‘infection’
Optional recovery to trial VPX while recovering MPX
Instructions for changing passwords
Instructions for re-keying certs
Instructions and guidance for LDAP password and AD policies to deny interactive logon to that account (note- using “Domain Users” on network shares is a bad idea for this reason)
Change Management
Schedule outage(s)
Ideally- schedule complete Gateway outage by firewall rule blockage
VPX – estimated recovery time
MPX – estimated recovery time
Consideration – remediate now, update/upgrade firmware later
Highly suggested that any person with Citrix Gateway access change their passwords
Download appropriate firmware
Change
During an outage and depending on your specific environment…
Run the remediation script
During a more extended outage
Backup appliance firmware or VM
Backup running/saved config
Applying the configuration to new firmware or updating a clean backup
The Great Password Reset of 2020
LDAP (we recommend changing it again)
Service Accounts
Any account that has accessed Citrix via the gateway should be changed
Note – there are instructions on the Citrix website for all of this, but I'm betting if you were confident about them – you wouldn't have read this far. The good news is I think I can help. Scroll down if that's you after you read the final step.
Maintain
Monitoring
Command line monitoring
Httpaccess logs
top
Citrix tool to test for compromise
Web Interface watching for policy hits
Taking future notices seriously
You have a day or less. This is proof. Not to get preachy here but if there's one thing this whole event proves
Getting notified of future issues
Regular Tasks to schedule
Change LDAP password every month after breach for 3 months
Change NSROOT password every 90 days
Internal Health Check – watch for .xml files, etc
Health Check by qualified consultants
Get it Done NOW
It is my opinion that you need to be confident to do everything on this list and do it right now, if you haven't already.
The way I see it you have three options:
Power thru it – spend the hours needed researching and making sure everything gets done. The good news is that there is plenty of information out there to help you get there- download the check list here.
Hire someone like me to help. By all means, I'm willing to help. But as you can imagine I'm quite busy. As of this writing my first availability is in early March. You'll need 4 hours minimum and I charge a minimum of $185USD per hour ($740).
Join my workshop on Feb. 17th. I'm taking a huge risk and setting aside a week, along with some of my friends and trusted advisors. We will walk you thru the process step by step and be available at regular times all week to answer your questions. The cost to join this workshop will be $399. If you are interested (or want your boss to pay for it) – see the contact form below and we'll get you in. Can't make it on the 17th? Don't worry – we'll be recording the sessions and I'll be available for office hours for your questions until March 30th, 2020 – and email after that (though hopefully you'd have fixed it by then!)
Grab my PDF Checklist
I put together a quick PDF document so you can make sure you've got your bases covered – download it here (no opt-in required)
Members of our email list were informed of this vulnerability before it became an issue. It is free to join and we'll keep you updated. Your email just goes to us - never to sponsors or anyone else.
My own opinions about this aside in terms of ethical hacking - a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC. UPDATE: I have put together a remediation...
I have learned a few things over the course of the last 20 years. These are things I've been teaching to other consultants, teams and coaching clients for years, so I've decided it was time to put them into a book!The book focuses on what we call a Methodology: a...
How much would you pay to get advice from 140 industry leaders? Great, now you know how much to donate. Because the download for this one is FREE. Download a PDF of the Byte-Sized Book. Or if you want to help out with One Laptop Per Child - donate, or buy the...
My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.
UPDATE: I have put together a remediation checklist and a few more details! You can download the checklist now – head to https://ctxpro.com/?p=1493 for more details.
Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.
Note the massive uptick in incidents. These people aren't doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won't end there so you really need to pay attention!
(please note – these are all external links and I can't validate any content in them because of that)
I will try to keep updates to this page as much as I am able – but for now please realize that until the dates below every NetScaler ADC is vulnerable without the code… and possibly even then.
I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours' for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.
What we know about CVE-2019-19781
The vulnerability affects all supported product versions and all supported platforms:
• Citrix ADC and Citrix Gateway version 13.0 all supported builds
• Citrix ADC and NetScaler Gateway version 12.1 all supported builds
• Citrix ADC and NetScaler Gateway version 12.0 all supported builds
• Citrix ADC and NetScaler Gateway version 11.1 all supported builds
• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds
Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.
Update 1.20.2020 – I'd like to add some additional POSSIBLE impacts, including the possibility that ALL accounts that have gone thru the Citrix Gateway should have their passwords changed right away in addition to other potential impacts. Please review Thomas's article for additional information.
One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you'll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks. No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don't be that person to use hurtful hashtags or spread fear rather than solutions.
Action Steps for CVE-2019-19781
Last update – 1.17.2020
Fix Script for Citrix NetScaler ADC
Use the instructions at https://support.citrix.com/article/CTX267679right now. The extended Citrix community is working on additional scripts. The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down' from the other, you need to replace it with one from your keyboard. Fortunately – the fix is quick but does require a reboot to take full effect.
UPDATE 1.17.2020
Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I'd do so now.
If someone is cryptomining on your ADC- you'll see high utilization – but there's a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.
Here's what do to. After you've run the prevention script and rebooted, get into the shell, or just enter
shell top -n 10
What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON'T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you're satisfied Ctrl-C will take you out of that.
However, in my mind, cryptomining is a secondary concern. Your company's information may have been exposed at some levels that have not yet been fully determined.
The big indicator of a compromise at this point is .xml files in directories they don't belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you've been exploited, exporting your configuration and configuring from fresh firmware isn't a bad idea.
If you are compromised:
Take the ADC off the network.
Change the password of any LDAP or other AD/network accounts stored on the ADC.
Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
Disconnect the network before starting
Start the machine and verify using the console that the VPX does not appear compromised
Change the nsroot password
Attach the internal network only
Run the fix (alternatively- type this via the console to be safer)
attach the external network
Keep an eye on the logs
Replace SSL Certificates on the appliance at your earliest opportunity
Timeline and Updates
Jan 13 2020 Citrix has announced a timeline for ADC firmware that will include fixes.
Note- these are the initial timelines superseded on Jan 19th
Version
Refresh Build
Expected Release Date
10.5
10.5.70.x
31st January 2020
11.1
11.1.63.x
20th January 2020
12.0
12.0.63.x
20th January 2020
12.1
12.1.55.x
27th January 2020
13.0
13.0.47.x
27th January 2020
Jan 14 2020
I have started working with clients to remediate compromises and double-check other clients. I've updated some suggested quick things above.
I'm tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!
…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.
DJ Eshelman announced that he will be working with a few other industry professionals to create a video and community support course offering to help people remediate this issue.
I have learned a few things over the course of the last 20 years. These are things I've been teaching to other consultants, teams and coaching clients for years, so I've decided it was time to put them into a book!
The book focuses on what we call a Methodology: a repeatable process for working that is predictable enough for others to recognize and learn. Rather than a complex process that requires a certifiation to even understand, I have found that successful organizations use a methodology that can be understood by everyone from sales, services, support and up to the C-Suite.
In other words- if you're looking for another book on Agile or other DevOps methods… while some of those can be implemented inside of this – I have found that these methods tend to isolate IT and isn't always effective. Why? Because if your entire team isn't on-board, it tends to fall apart. So what happens when someone new comes in? Or a key person leaves? Or… the team gets lax in taking actions? What happens when moving too fast causes outages from risks that weren't properly identified? I call it a Resume Generating Opportunity. And it is exactly what I want to see people avoid.
That's why I have kept this book outside of the theoretical and describe EXACTLY what works NOW, and has been easy to understand everywhere I have taken it. The book is helpful if you are in sales, a service desk or a seasoned consultant. Everyone has something they can learn and I have plenty to teach!
I need your help! This survey will help me determine the best title and keywords to use to make sure everyone sees this book who needs it! I'd very much appreciate your vote!
There is a lot of marketing, hype and FUD (Fear, Uncertainty & Doubt) thrown around in the EUC (End User Computing) space. So I'm always grateful for true survey results to see the true trends in the industry. So, let's have a look!
My friends Mark Plettenberg, Ruben Spruijt and Christiaan Brinkhoff have compiled this 73-page report that speaks the truth about the trends in VDI and EUC in general. Worldwide, just under 600 people turned in surveys which the team decoded and compared to previous years. Here are some of my personal favorite highlights.
Highlights of the 2019 State of the Union
Healthcare continues to be the largest business vertical staying on-premises for EUC.
Age of environments is interesting – VDI environments are often exceeding 5 years without significant updates. New designs had been declining but saw an uptick this year.
Citrix still dominates the VDI market but decreased by 7% to just over 50% of survey responses this year.
Nearly 22% are still using Windows 7
FSLogix use increased by about 2% – note that this was probably before people really knew they can get it for free now… But Citrix UPM is still the most popular choice.
PVS is still popular for imaging though usage is declining. MCS is almost neck-and-neck with PVS.
39% of respondents were from North America with 50 overall countries represented.
The majority of companies have between 1,000 and 4,999 users
An overwhelming percentage (76%) are still using On-Premises Server Based Computing (RDSH or Multi-Session OS) – this only decreased by 2%.
VMware vSphere is still over 50% of the market, however, the big note I took from it was that Citrix, Microsoft and Nutanix Hypervisors are all increasing while VMware is decreasing. KVM (driven mostly by Nutanix) is up significantly this year.
More and more people continue to think of VDI as stateless or non-persistent.
5.6% of respondents say they use Citrix EdgeSight for monitoring. If that doesn't shock you, it's overall position on the chart will. Not bad for being dead for nearly a decade.
I could go on- but honestly, I think you should download the guide and compare your results!
