First things first – don't panic if you are one of the literally thousands of companies that were affected by CVE-2019-19781… because everyone was! But just because I say don't panic doesn't mean do nothing. I (DJ) have been working with several clients, experts and Citrix around solutions and I think we are in a place where most people have at least stopped the bleeding… but I grow concerned that many more have not.
Every. Single. Citrix ADC (NetScaler) was vulnerable and should be assumed to have been a target.
Some sentences are really hard to write. That last one was definitely one of them. But that all pales in comparison to the implications. The reality is- not only were several thousand companies not ready to handle this threat, my industry friends remain concerned that several thousand customers have not remediated because they didn't know how. Citrix and several sites including my own have given instructions – but if you don't know a few key things about the ADC and how to manage it outside of the GUI (Web Interface) then there is a confidence problem that has kept people from fixing the issue out of fear of locking out their users or making matters worse.
This has kept me busy at home and at hotel rooms for the last few months and I've come to realize I don't have the ability to help everyone. Here's what I can tell you:
- Citrix ADC (NetScaler) VPX with a backup before December has been able to be patched and remediation performed in about 3 consulting hours.
- Citrix ADC VPX with no backups have typically taken between 4 and 5 consulting hours.
- Citrix ADC MPX (hardware platforms) are a particular challenge and in some cases have taken several evenings of downtime to resolve.
- The skills needed to properly assess breaches, prevent them and fully remediate from the real threats of CVE-2019-19781 are those that are not commonly used, even by those that administrate Citrix environments.
- Many customers deployed Citrix ADCs as a replacement to legacy Secure Gateways and have zero skills administrating NetScaler/ADCs, depending on the contractors that deployed them – contractors that often have moved on or are now so busy with these efforts on top of their existing workloads that it has become impossible to schedule help for many of them.
- Worst of all – even though this has been a very popular and well documented case (my article “Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?” is the most popular article I've ever had. It eclipsed a nearly 3 year old popular article in a few weeks) many with Citrix NetScalers don't even know about the issue.
To help, I've developed a kind of a checklist/lesson plan, using my RiskLESS Methodology. Expand each section to see the detail of what needs to be done and the tools and knowledge you will need to be aware of to make it happen.
Understand
Understand the Threat
History of the issue
What is important to know about how the ADC works in this regard
What Hackers are doing (known threats)
What Citrix is Doing
Additional considerations
- LDAP threats – high potential for network compromise and backdoors being setup up undetected
- If LDAP account was also domain or elevated admin – critical to change password but also begin changing other domain accounts
- SSL Key decode threats and need to re-key
- NSROOT and any other local account passwords need to be changed
Detection
Citrix Tools
- How to install the detection tool & use it
- Web Admin Interface
- Insight Services (CIS)
- Shell bash scripts and python scripts
Command Line Inspection
- Install PuTTY and SCP
- Using SCP to download config + logs
- Gathering breach info using the Citrix/FireEye Tool.
Documentation
Written Documentation
- What you saw, when
- Screenshots, etc
- Document your findings of the exposure to management
- Collect Passwords and store securely
- NSROOT
- LDAP account
- myCitrix account
- SSL provider
Remediation Considerations
- Understand the remediation differences between VPX, MPX and SDX
- If VPX/SDX – see if there are recoverable backups from before Dec 17th 2019 (if so, celebrate)
- HA and Cluster considerations
- Note to check all nodes for compromise – may make recovery easier
- Firmware version considerations
- Some versions of the firmware the remediation script did not work
- Disclosure of breach and description to management
- If LDAP account was elevated, severe danger requiring rapid disclosure to management and legal
Plan
Fast Remediation Steps
Determine what steps to take immediately and what will wait for a longer change window if this will be required (ie, MPX when access to the gateway will be needed during change window)
- Emergency change considerations – brief outage for reboot
- Apply remediation script and reboot
- Create temp LDAP account password
- Change LDAP account and/or password
Written Plan for Extended Remediation
- Write out a plan of action based on your platform
- Instructions for VPX
- clean install
- Plan for VPX recovery
- Plan for HA secondary clean upgrade + fresh install primary
- Instructions for SDX
- Instructions for MPX
- Special consideration for firmware ‘infection’
- Optional recovery to trial VPX while recovering MPX
- Instructions for changing passwords
- Instructions for re-keying certs
- Instructions and guidance for LDAP password and AD policies to deny interactive logon to that account (note- using “Domain Users” on network shares is a bad idea for this reason)
- Instructions for VPX
Change Management
- Schedule outage(s)
- Ideally- schedule complete Gateway outage by firewall rule blockage
- VPX – estimated recovery time
- MPX – estimated recovery time
- Consideration – remediate now, update/upgrade firmware later
- Communication plan (with users, management, IT)
- Schedule staff or Consultants involved
Generate and store new passwords
- USE A PASSWORD MANAGER – I use Dashlane
- Any compromised password needs to be changed
- Highly suggested that any person with Citrix Gateway access change their passwords
Download appropriate firmware
Change
During an outage and depending on your specific environment…
- Run the remediation script
During a more extended outage
- Backup appliance firmware or VM
- Backup running/saved config
- Applying the configuration to new firmware or updating a clean backup
- The Great Password Reset of 2020
- LDAP (we recommend changing it again)
- Service Accounts
- Any account that has accessed Citrix via the gateway should be changed
Note – there are instructions on the Citrix website for all of this, but I'm betting if you were confident about them – you wouldn't have read this far. The good news is I think I can help. Scroll down if that's you after you read the final step.
Maintain
Monitoring
- Command line monitoring
- Httpaccess logs
- top
- Citrix tool to test for compromise
- Web Interface watching for policy hits
Taking future notices seriously
You have a day or less. This is proof. Not to get preachy here but if there's one thing this whole event proves
Getting notified of future issues
Regular Tasks to schedule
- Change LDAP password every month after breach for 3 months
- Change NSROOT password every 90 days
- Internal Health Check – watch for .xml files, etc
- Health Check by qualified consultants
Get it Done NOW
It is my opinion that you need to be confident to do everything on this list and do it right now, if you haven't already.
The way I see it you have three options:
- Power thru it – spend the hours needed researching and making sure everything gets done. The good news is that there is plenty of information out there to help you get there- download the check list here.
- Hire someone like me to help. By all means, I'm willing to help. But as you can imagine I'm quite busy. As of this writing my first availability is in early March. You'll need 4 hours minimum and I charge a minimum of $185USD per hour ($740).
- Join my workshop on Feb. 17th. I'm taking a huge risk and setting aside a week, along with some of my friends and trusted advisors. We will walk you thru the process step by step and be available at regular times all week to answer your questions. The cost to join this workshop will be $399. If you are interested (or want your boss to pay for it) – see the contact form below and we'll get you in. Can't make it on the 17th? Don't worry – we'll be recording the sessions and I'll be available for office hours for your questions until March 30th, 2020 – and email after that (though hopefully you'd have fixed it by then!)
Grab my PDF Checklist
I put together a quick PDF document so you can make sure you've got your bases covered – download it here (no opt-in required)
Contact us for Help
Related Articles
Citrix 2023: What Changed?
Citrix has changed a bit in 4 years. So, what would D.J. do differently if Be a Citrix Hero were written in 2023?
Career Choices
So you've decided maybe you like this whole #Citrix thing. What are your #Career options? Join DJ Eshelman as he walks you thru his 4 (of 6) basic Success Lanes for IT roles in Citrix. What roles do you see yourself being most successful in? We'd love to hear from you...
A Simple Services Methodology
In this episode, DJ explains his Services Methodology which will be featured in his new book "Just Do This" (summer of 2020). You'll learn how DJ has taken the methodology from companies like Microsoft, Citrix and others, combined it with some psychology (and...
How Much Hair Should DJ Cut Off? You Decide!
Admit it, you've wondered when DJ will cut his hair. The answer is April 10th. How much hair he cuts is up to YOU – find out how to participate!
Windows 11 May Challenge Performance Tuning for VDI
The way System Engineers approach VDI performance and user perception looks to be changing dramatically in Windows 11. Here's what we know so far.
Top 10 Tips for Surviving Tech Conferences (like Citrix Synergy)
In this special edition - we're using Citrix Synergy as the perfect excuse to talk about my top 10 tips for surviving a conference! Here's the summary of the 10 Tips I talk about in this video: 1- Remember - We're all here for the same reason 2- Give Yourself Space 3-...
