fbpx
Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

February 6, 2020

First things first – don't panic if you are one of the literally thousands of companies that were affected by CVE-2019-19781… because everyone was! But just because I say don't panic doesn't mean do nothing. I (DJ) have been working with several clients, experts and Citrix around solutions and I think we are in a place where most people have at least stopped the bleeding… but I grow concerned that many more have not.

Every. Single. Citrix ADC (NetScaler) was vulnerable and should be assumed to have been a target.

Some sentences are really hard to write. That last one was definitely one of them. But that all pales in comparison to the implications. The reality is- not only were several thousand companies not ready to handle this threat, my industry friends remain concerned that several thousand customers have not remediated because they didn't know how. Citrix and several sites including my own have given instructions – but if you don't know a few key things about the ADC and how to manage it outside of the GUI (Web Interface) then there is a confidence problem that has kept people from fixing the issue out of fear of locking out their users or making matters worse.

This has kept me busy at home and at hotel rooms for the last few months and I've come to realize I don't have the ability to help everyone. Here's what I can tell you:

  • Citrix ADC (NetScaler) VPX with a backup before December has been able to be patched and remediation performed in about 3 consulting hours.
  • Citrix ADC VPX with no backups have typically taken between 4 and 5 consulting hours.
  • Citrix ADC MPX (hardware platforms) are a particular challenge and in some cases have taken several evenings of downtime to resolve.
  • The skills needed to properly assess breaches, prevent them and fully remediate from the real threats of CVE-2019-19781 are those that are not commonly used, even by those that administrate Citrix environments.
  • Many customers deployed Citrix ADCs as a replacement to legacy Secure Gateways and have zero skills administrating NetScaler/ADCs, depending on the contractors that deployed them – contractors that often have moved on or are now so busy with these efforts on top of their existing workloads that it has become impossible to schedule help for many of them.
  • Worst of all – even though this has been a very popular and well documented case (my article “Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?” is the most popular article I've ever had. It eclipsed a nearly 3 year old popular article in a few weeks) many with Citrix NetScalers don't even know about the issue.

To help, I've developed a kind of a checklist/lesson plan, using my RiskLESS Methodology. Expand each section to see the detail of what needs to be done and the tools and knowledge you will need to be aware of to make it happen.

Understand

Understand the Threat

History of the issue

What is important to know about how the ADC works in this regard

What Hackers are doing (known threats)

What Citrix is Doing

Additional considerations

  • LDAP threats – high potential for network compromise and backdoors being setup up undetected
  • If LDAP account was also domain or elevated admin – critical to change password but also begin changing other domain accounts
  • SSL Key decode threats and need to re-key
  • NSROOT and any other local account passwords need to be changed

Detection

Citrix Tools
  • How to install the detection tool & use it
  • Web Admin Interface
  • Insight Services (CIS)
  • Shell bash scripts and python scripts
Command Line Inspection
  • Install PuTTY and SCP
  • Using SCP to download config + logs
  • Gathering breach info using the Citrix/FireEye Tool.

Documentation

Written Documentation
  • What you saw, when
  • Screenshots, etc
  • Document your findings of the exposure to management
  • Collect Passwords and store securely
    • NSROOT
    • LDAP account
    • myCitrix account
    • SSL provider
Remediation Considerations
  • Understand the remediation differences between VPX, MPX and SDX
  • If VPX/SDX – see if there are recoverable backups from before Dec 17th 2019 (if so, celebrate)
  • HA and Cluster considerations
  • Note to check all nodes for compromise – may make recovery easier
  • Firmware version considerations
  • Some versions of the firmware the remediation script did not work
  • Disclosure of breach and description to management
  • If LDAP account was elevated, severe danger requiring rapid disclosure to management and legal
Plan

Fast Remediation Steps

Determine what steps to take immediately and what will wait for a longer change window if this will be required (ie, MPX when access to the gateway will be needed during change window)

  • Emergency change considerations – brief outage for reboot
  • Apply remediation script and reboot
  • Create temp LDAP account password
  • Change LDAP account and/or password

Written Plan for Extended Remediation

  • Write out a plan of action based on your platform
    • Instructions for VPX
      • clean install
      • Plan for VPX recovery
      • Plan for HA secondary clean upgrade + fresh install primary
    • Instructions for SDX
    • Instructions for MPX
      • Special consideration for firmware ‘infection’
      • Optional recovery to trial VPX while recovering MPX
    • Instructions for changing passwords
    • Instructions for re-keying certs
    • Instructions and guidance for LDAP password and AD policies to deny interactive logon to that account (note- using “Domain Users” on network shares is a bad idea for this reason)

Change Management

  • Schedule outage(s)
    • Ideally- schedule complete Gateway outage by firewall rule blockage
    • VPX – estimated recovery time
    • MPX – estimated recovery time
    • Consideration – remediate now, update/upgrade firmware later
  • Communication plan (with users, management, IT)
  • Schedule staff or Consultants involved

Generate and store new passwords

  • USE A PASSWORD MANAGER – I use Dashlane
  • Any compromised password needs to be changed
  • Highly suggested that any person with Citrix Gateway access change their passwords

Download appropriate firmware

Change

During an outage and depending on your specific environment…

  • Run the remediation script

 During a more extended outage

  • Backup appliance firmware or VM
  • Backup running/saved config
  • Applying the configuration to new firmware or updating a clean backup
  • The Great Password Reset of 2020
    • LDAP (we recommend changing it again)
    • Service Accounts
    • Any account that has accessed Citrix via the gateway should be changed

Note – there are instructions on the Citrix website for all of this, but I'm betting if you were confident about them – you wouldn't have read this far. The good news is I think I can help. Scroll down if that's you after you read the final step.

Maintain

Monitoring

  • Command line monitoring
    • Httpaccess logs
    • top
  • Citrix tool to test for compromise
  • Web Interface watching for policy hits

Taking future notices seriously

You have a day or less. This is proof. Not to get preachy here but if there's one thing this whole event proves

Getting notified of future issues

Regular Tasks to schedule

  • Change LDAP password every month after breach for 3 months
  • Change NSROOT password every 90 days
  • Internal Health Check – watch for .xml files, etc
  • Health Check by qualified consultants

Get it Done NOW

It is my opinion that you need to be confident to do everything on this list and do it right now, if you haven't already.

The way I see it you have three options:

  1. Power thru it – spend the hours needed researching and making sure everything gets done. The good news is that there is plenty of information out there to help you get there- download the check list here.
  2. Hire someone like me to help. By all means, I'm willing to help. But as you can imagine I'm quite busy. As of this writing my first availability is in early March. You'll need 4 hours minimum and I charge a minimum of $185USD per hour ($740).
  3. Join my workshop on Feb. 17th. I'm taking a huge risk and setting aside a week, along with some of my friends and trusted advisors. We will walk you thru the process step by step and be available at regular times all week to answer your questions. The cost to join this workshop will be $399. If you are interested (or want your boss to pay for it) – see the contact form below and we'll get you in. Can't make it on the 17th? Don't worry – we'll be recording the sessions and I'll be available for office hours for your questions until March 30th, 2020 – and email after that (though hopefully you'd have fixed it by then!)

Grab my PDF Checklist

I put together a quick PDF document so you can make sure you've got your bases covered – download it here (no opt-in required)

Contact us for Help

Related Articles

Grab Your Copy of the Byte-Sized Book

How much would you pay to get advice from 140 industry leaders? Great, now you know how much to donate. Because the download for this one is FREE. Download a PDF of the Byte-Sized Book. Or if you want to help out with One Laptop Per Child - donate, or buy the...

Proactive App Compatibility Testing is now Free with Login AT Express

Proactive App Compatibility Testing is now Free with Login AT Express

Those of you that have read a lot of what I write know that I am very much about being proactive in IT. Proactive means you are aware of issues before they affect work or worse, production. One of the major risk areas I have been seeing, however, is when applications are not tested against system changes or validated prior to being placed into a Citrix solution. Teams are forced into being reactive. And that's bad.

Reactive = Bad

What is happening here is that IT is RE-active not Proactive. Today we're going to look quickly at two primary problem areas and I'll tell you about some friends that are trying to help.

That Compatibility Problem

I'll be bold here and say that one thing that seriously irks me is when people just assume an application will work virtually – often without any testing in a physical realm. I've lost count of how many times this year I have heard teams frustrated because there is an initiative to upgrade to Windows 10 but management just assumed the apps on their Windows 7 (or Server 2008 – not R2) environment would ‘just work'. That's bad.

Without good proactive tools, what ends up happening is that IT just tries to make it work and spends countless hours when it doesn't. The worst part is that they typically don't even know it doesn't work until they are into the validation phase of a project, in most cases after several months have already been spent building the new environment. That's really bad.

That Patch Tuesday Problem

What we *haven't* yet addressed is updates. The other area that often causes a scramble is testing patches against the system. Problem as in… it is rarely done at this point. There may be a quick validation process before patches go out in maybe 1/3 of the customers I visit.

The vast majority of IT Teams managing Citrix are so overwhelmed at this point they rarely even test. They just deploy and hope for the best. They quite literally deploy and cross their fingers, knowing they'll just react when the trouble tickets come in.

How Reactive Affects Work

The reality is that IT is being hamstrung before they even start. How can they address a backlog of work when they are constantly reacting to new issues or having to re-engineer solutions because they didn't have the info they needed in advance?

Proactive = Good

So maybe I'm talking more to management here – what your IT team needs here are two things:

  1. Proactive Mindset of Management (solution AFTER problem, not before it)
  2. Proactive Tools (force multipliers with automation)

Proactive Mindset

When you are constantly chasing the latest issue, there's only so much time in the day. Improving becomes relegated to lower and lower on the list, and in no time at all the team is only ever break-fixing. This problem has become so extreme that I have seen IT teams split between deployment and administration tasks completely, often trading roles mid-year just to retain sanity. Others have teams that don't specialize at all, they just deploy and are never tasked with upkeep.

The problem here is that both of these solutions make innovation either not happen at all, or be so disconnected with user needs as to become worthless. I believe that the IT team supporting an environment needs to be in some way committed to improving it. Both in terms of them knowing the issues very closely and just raw sanity. Most folks in EUC do not want to spend most of their time doing only one thing. They want to grow. And this is where having a completely reactive mindset makes such things impossible.

What is required here is a cultural shift in the workplace to identify issues before they become production outages or “Resume Generating Opportunities”. Management needs to drive this – not the IT teams. Leadership must lead. There; I said it now I'll jump off my soapbox and we can continue.

Once the mindset is set towards being proactive instead of reactive – workflows need to change to accommodate it. What one quickly realizes is that being proactive requires an investment of time and faith. The time is obvious, but the faith is knowing that historically, teams that invest proactive time spend less overall time troubleshooting issues and chasing outages. It is easy to forget this and take it as a given, falling back into old habits. Mind the new culture!

Proactive Tools

But today I have some good news. Automation can be a ‘force multiplier' for IT to accomplish the same goals in less time, especially with repeatable tasks.

Login AT from LoginVSI is an automated testing tool for applications that can tell you if applications are going to be able to run in a scenario… without the difficulty of maintaining AppDNA or ‘relying on the testing army' to give proactive results before deployment. The software is able to automatically perform key tests against the software in a consistent manner (something that is often missed relying on human testing) and inform you in advance of potential trouble.

By doing this proactively and automatically, you can save your team a lot of wasted effort hours in finding the same thing.

Login AT Express = Free

I love the attitude at LoginVSI. If you've followed me for any length of time you know that I'm on a mission to Leave the World Better Than I Found It.

Here's some text from LoginVSI's press release about Login AT Express:

Current Login AT customers are very large enterprises with very large numbers of applications. In the past year we learned that small and medium sized enterprises are also suffering the same problem.
As of the start Login VSI has always been active to help ‘make the VDI world a better place’. With our product Login VSI as a free tool for the first years, our support of VDIlikeaPRO, the performance tests in our own lab and the active support of 3rd party labs with free licenses (such as ICT-R and others).
With so many organizations struggling to keep their applications running after each software update, and our relatively small organization to effectively help them all, we chose to adopt the freemium distribution model for this new product to enable us to reach the market in the most efficient way.

LoginVSI Login AT Express Press Release

I'm happy to announce that Login AT Express is now free for up to 50 applications. All you need to do is renew your 1 month license as often as you like.

I'm willing to bet this will benefit well over 3/4 of my audience, so I'm thrilled about this announcement.

Let's get proactive and stop the grind!

Here's what to do. Click the button below and fill out the form. Under Product of Interest, make sure you select Login AT Express (and the others). Do me a favor and let them know you heard it from ctxpro.com as a blogpost!

Earnings Disclosure: DJ Eshelman is a LoginVSI Technology Advocate and was not compensated in any way for this article or any links associated with it.

How to Press On When Noone Appreciates Your Work

How to Press On When Noone Appreciates Your Work

How to Press On When Noone Appreciates Your Work

May 16, 2019

You know how Facebook has that whole flashback/timehop “X years ago today” feature? How cool would it be to have that in the IT world?

“3 years ago today you said you were going to quit because your Citrix session took 5 minutes to load…”

I had a reminder of this recently when I was doing some work at a client having a comparatively minor problem. When I was called in initially, they had staff wanting to quit.

I'm not joking. In fact I've encountered this more than once where staff were literally threatening to resign their means of working because they were so frustrated in how their employer was making them work. And to them- it was a “Citrix” problem. These guys were, however, especially desperate. Things were really, really bad. Like 12 minute logon bad.

Now they are one of my favorite success stories for why you should apply leading practices. In their case it literally kept staff from quitting because they couldn't work. Surely it can't be pure irony that a month later another Citrix environment I served had a similar issue. It was like a theme for that year.

I found myself wishing there was some way to give perspective to those we serve about just how much better things are without just bragging about it. Timehop for EUC (but, you know, without the data breaches). Flashback for Citrix.

But that got me thinking… why do I do this stuff. I mean, why really?

For all of you facing another day with more tickets, complaints and a seemingly never-ending series of providing praise-less solutions – I'll simply ask you:

Are you *really* doing it for the money?

Or for recognition?

Both of those will fail you.

I encourage you to connect with #WHY you are doing it. If you're fine-tuning Citrix for a 5 second faster logon – is it the achievement that drives you, perhaps?

This too will pass unnoticed.

I feel my mission in life comes down to a simple statement- one that actually resonated with former Citrix CEO Mark Templeton.

“In whatever you do, Leave it Better Than You Found It”

Why do heroes do what they do? Why endure hardships, pain and rejection… why live a solitary life?

Because that's what heroes do. They leave what they found better. Not because they are obligated but because it is their mission or that they are seeking praise. They just can't help it – it's how they are wired. Maybe they were born that way. But most have been forged from circumstance and hardship, and emerged with an overwhelming sense of one thing:

All that matters is that I have mattered. Therefore I must endeavor to Leave it Better than I Found It.

This may not be you. And that's okay. But I submit to you that if you really want to advance in your career it is more about serving than attaining. More about leading than advancing.

In the coming months, I will be opening up opportunities for learning from myself and others just how to do this. Until then, consider your own motivations…

We may not be the hero they always deserve. But we *are* the Hero they *need* right now.

 

I don't do this for the praise.

I do this because I KNOW I've made their life better.

Whether they choose to recognize that or not really isn't the point in #EUC (End User Computing). The point is that, as my friend Dane Young reminded me of recently…. sometimes we are a lot more like Batman. We may not be the hero they always deserve.

But we *are* the hero they *need* right now.

So step up. Be the Hero.

So what can you do? Here's a few tips to get you started.

  • Connect with WHY you are working. Not why you are doing what you are doing… why you are WORKING.
  • Make sure WHAT you are doing matches WHY you want to do it.
  • Serving others should be your primary focus in life.
  • Spend more effort learning how to be a better person than to be a better tech
  • CELEBRATE the wins you have
  • Don't dwell on failures
  • Be INCLUSIVE instead of exclusive
  • Find a mentor

By the way- if you want to be coached by me – “TheCitrixCoach” isn't just a fancy title. I do actually coach individuals and teams on how to improve their careers. It isn't just teaching Citrix skills. It's about going on the journey together!

Join the Conversation

Related Articles

Grab Your Copy of the Byte-Sized Book

How much would you pay to get advice from 140 industry leaders? Great, now you know how much to donate. Because the download for this one is FREE. Download a PDF of the Byte-Sized Book. Or if you want to help out with One Laptop Per Child - donate, or buy the...

Stay Up to Date With The Latest News & Updates

Access Premium Content

Join our Citrix Hero Program and get new lessons every month!

Get updates right to your inbox

Curated news from the perspective of a Citrix Expert, not a sales person… delivered right to you.

Follow The Leaders

Making technology fun again…

Why I Gave Away My Top Citrix Tips

“DJ, why are you giving away your best Citrix Tips?”

This question was asked of me recently- and I didn't hesitate to give an answer. So I think I'll give it to everyone here as a kind of open letter to the Citrix community…

It all has to do with quantity. I seem to give out the same things almost every week, not only in conversation but in my consulting engagements as well (regardless of if I'm representing Citrix, a Reseller or my own consulting company). So I asked myself two questions:

  1. Why do I always seem to be giving the same advice? Is that such a bad thing? It makes my job easier, right?
  2. Are they really that important if people aren't doing them?

Before I get into this behind-the-scenes look (or if you don't really want to know why, you just want my Top 3 tips right now) – click here to register for my free eBook where I give you those and put you on a monthly newsletter with even more tips – free:

Register for the Free eBook & top Citrix tips

(more…)

Proactive App Compatibility Testing is now Free with Login AT Express

Going Beyond CYA – Why I Secure Passwords with Dashlane

I started off the day with an extortion letter. Someone managed to grab a password I used 10+ years ago. This person (or possibly even company/country, this email was surprisingly well written) thinks that I'll wire them 7000 USD in Bitcoin to keep them from sending supposedly ‘incriminating' videos to my contacts…. because they have access to my computer and my webcam… yeah, sure.
Clever- but not going to happen. They don't have access to my computer, but they do have my attention!

(more…)

Categories

By With a Little Help from Our Friends

ByteSized Book logo