fbpx
Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

February 6, 2020

First things first – don't panic if you are one of the literally thousands of companies that were affected by CVE-2019-19781… because everyone was! But just because I say don't panic doesn't mean do nothing. I (DJ) have been working with several clients, experts and Citrix around solutions and I think we are in a place where most people have at least stopped the bleeding… but I grow concerned that many more have not.

Every. Single. Citrix ADC (NetScaler) was vulnerable and should be assumed to have been a target.

Some sentences are really hard to write. That last one was definitely one of them. But that all pales in comparison to the implications. The reality is- not only were several thousand companies not ready to handle this threat, my industry friends remain concerned that several thousand customers have not remediated because they didn't know how. Citrix and several sites including my own have given instructions – but if you don't know a few key things about the ADC and how to manage it outside of the GUI (Web Interface) then there is a confidence problem that has kept people from fixing the issue out of fear of locking out their users or making matters worse.

This has kept me busy at home and at hotel rooms for the last few months and I've come to realize I don't have the ability to help everyone. Here's what I can tell you:

  • Citrix ADC (NetScaler) VPX with a backup before December has been able to be patched and remediation performed in about 3 consulting hours.
  • Citrix ADC VPX with no backups have typically taken between 4 and 5 consulting hours.
  • Citrix ADC MPX (hardware platforms) are a particular challenge and in some cases have taken several evenings of downtime to resolve.
  • The skills needed to properly assess breaches, prevent them and fully remediate from the real threats of CVE-2019-19781 are those that are not commonly used, even by those that administrate Citrix environments.
  • Many customers deployed Citrix ADCs as a replacement to legacy Secure Gateways and have zero skills administrating NetScaler/ADCs, depending on the contractors that deployed them – contractors that often have moved on or are now so busy with these efforts on top of their existing workloads that it has become impossible to schedule help for many of them.
  • Worst of all – even though this has been a very popular and well documented case (my article “Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?” is the most popular article I've ever had. It eclipsed a nearly 3 year old popular article in a few weeks) many with Citrix NetScalers don't even know about the issue.

To help, I've developed a kind of a checklist/lesson plan, using my RiskLESS Methodology. Expand each section to see the detail of what needs to be done and the tools and knowledge you will need to be aware of to make it happen.

Understand

Understand the Threat

History of the issue

What is important to know about how the ADC works in this regard

What Hackers are doing (known threats)

What Citrix is Doing

Additional considerations

  • LDAP threats – high potential for network compromise and backdoors being setup up undetected
  • If LDAP account was also domain or elevated admin – critical to change password but also begin changing other domain accounts
  • SSL Key decode threats and need to re-key
  • NSROOT and any other local account passwords need to be changed

Detection

Citrix Tools
  • How to install the detection tool & use it
  • Web Admin Interface
  • Insight Services (CIS)
  • Shell bash scripts and python scripts
Command Line Inspection
  • Install PuTTY and SCP
  • Using SCP to download config + logs
  • Gathering breach info using the Citrix/FireEye Tool.

Documentation

Written Documentation
  • What you saw, when
  • Screenshots, etc
  • Document your findings of the exposure to management
  • Collect Passwords and store securely
    • NSROOT
    • LDAP account
    • myCitrix account
    • SSL provider
Remediation Considerations
  • Understand the remediation differences between VPX, MPX and SDX
  • If VPX/SDX – see if there are recoverable backups from before Dec 17th 2019 (if so, celebrate)
  • HA and Cluster considerations
  • Note to check all nodes for compromise – may make recovery easier
  • Firmware version considerations
  • Some versions of the firmware the remediation script did not work
  • Disclosure of breach and description to management
  • If LDAP account was elevated, severe danger requiring rapid disclosure to management and legal
Plan

Fast Remediation Steps

Determine what steps to take immediately and what will wait for a longer change window if this will be required (ie, MPX when access to the gateway will be needed during change window)

  • Emergency change considerations – brief outage for reboot
  • Apply remediation script and reboot
  • Create temp LDAP account password
  • Change LDAP account and/or password

Written Plan for Extended Remediation

  • Write out a plan of action based on your platform
    • Instructions for VPX
      • clean install
      • Plan for VPX recovery
      • Plan for HA secondary clean upgrade + fresh install primary
    • Instructions for SDX
    • Instructions for MPX
      • Special consideration for firmware ‘infection’
      • Optional recovery to trial VPX while recovering MPX
    • Instructions for changing passwords
    • Instructions for re-keying certs
    • Instructions and guidance for LDAP password and AD policies to deny interactive logon to that account (note- using “Domain Users” on network shares is a bad idea for this reason)

Change Management

  • Schedule outage(s)
    • Ideally- schedule complete Gateway outage by firewall rule blockage
    • VPX – estimated recovery time
    • MPX – estimated recovery time
    • Consideration – remediate now, update/upgrade firmware later
  • Communication plan (with users, management, IT)
  • Schedule staff or Consultants involved

Generate and store new passwords

  • USE A PASSWORD MANAGER – I use Dashlane
  • Any compromised password needs to be changed
  • Highly suggested that any person with Citrix Gateway access change their passwords

Download appropriate firmware

Change

During an outage and depending on your specific environment…

  • Run the remediation script

 During a more extended outage

  • Backup appliance firmware or VM
  • Backup running/saved config
  • Applying the configuration to new firmware or updating a clean backup
  • The Great Password Reset of 2020
    • LDAP (we recommend changing it again)
    • Service Accounts
    • Any account that has accessed Citrix via the gateway should be changed

Note – there are instructions on the Citrix website for all of this, but I'm betting if you were confident about them – you wouldn't have read this far. The good news is I think I can help. Scroll down if that's you after you read the final step.

Maintain

Monitoring

  • Command line monitoring
    • Httpaccess logs
    • top
  • Citrix tool to test for compromise
  • Web Interface watching for policy hits

Taking future notices seriously

You have a day or less. This is proof. Not to get preachy here but if there's one thing this whole event proves

Getting notified of future issues

Regular Tasks to schedule

  • Change LDAP password every month after breach for 3 months
  • Change NSROOT password every 90 days
  • Internal Health Check – watch for .xml files, etc
  • Health Check by qualified consultants

Get it Done NOW

It is my opinion that you need to be confident to do everything on this list and do it right now, if you haven't already.

The way I see it you have three options:

  1. Power thru it – spend the hours needed researching and making sure everything gets done. The good news is that there is plenty of information out there to help you get there- download the check list here.
  2. Hire someone like me to help. By all means, I'm willing to help. But as you can imagine I'm quite busy. As of this writing my first availability is in early March. You'll need 4 hours minimum and I charge a minimum of $185USD per hour ($740).
  3. Join my workshop on Feb. 17th. I'm taking a huge risk and setting aside a week, along with some of my friends and trusted advisors. We will walk you thru the process step by step and be available at regular times all week to answer your questions. The cost to join this workshop will be $399. If you are interested (or want your boss to pay for it) – see the contact form below and we'll get you in. Can't make it on the 17th? Don't worry – we'll be recording the sessions and I'll be available for office hours for your questions until March 30th, 2020 – and email after that (though hopefully you'd have fixed it by then!)

Grab my PDF Checklist

I put together a quick PDF document so you can make sure you've got your bases covered – download it here (no opt-in required)

Contact us for Help

Related Articles

Be A Citrix Hero During the Coronavirus COVID-19 Outbreak

Be A Citrix Hero During the Coronavirus COVID-19 Outbreak

Whether you consider the responses thus far to “Coronavirus” to be panic or sensible precautions, one thing is sure: There hasn’t been a sudden demand for working from home worldwide like this … So how can you be the Citrix Hero during a season like this?
In this article we explore how D.J. Eshelman is responding to the increased demands on Citrix environments.

Install Teams and OneNote Per-Machine

Install Teams and OneNote Per-Machine

Microsoft Teams has been a bit of a nightmare for a long time. This is because Microsoft wasn't following Microsoft's own rules and was installing the app per-user... in AppData! This is a profile management nightmare and always has been. The story is pretty much the...

Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?

My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.

UPDATE: I have put together a remediation checklist and a few more details! You can download the checklist now – head to https://ctxpro.com/?p=1493 for more details.

Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.

Fun little hacking for posturing is one thing. But now that money is involved, we see just how unethical this hacking really is. Manuel Kalloff has a great summary of the true nature of these impacts here: https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/

Note the massive uptick in incidents. These people aren't doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won't end there so you really need to pay attention!

You can also track this event here: https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/

(please note – these are all external links and I can't validate any content in them because of that)

I will try to keep updates to this page as much as I am able – but for now please realize that until the dates below every NetScaler ADC is vulnerable without the code… and possibly even then.

Update 1.20.2020

Citrix has increased their efforts on this remediation with new firmware. Initial releases are out now, with key releases like 12.1 following on Jan 24th. Blog with details here: https://www.citrix.com/blogs/2020/01/19/vulnerability-update-first-permanent-fixes-available-timeline-accelerated/

Update 1.18.2020

I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours' for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.

What we know about CVE-2019-19781

The vulnerability affects all supported product versions and all supported platforms:

• Citrix ADC and Citrix Gateway version 13.0 all supported builds

• Citrix ADC and NetScaler Gateway version 12.1 all supported builds

• Citrix ADC and NetScaler Gateway version 12.0 all supported builds

• Citrix ADC and NetScaler Gateway version 11.1 all supported builds

• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds

https://support.citrix.com/article/CTX267027
Update Jan 17 2020 – ” This vulnerability also affects certain deployments of two older versions of our Citrix SD-WAN WANOP product versions 10.2.6 and version 11.0.3 ” from https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

What this Citrix ADC Vulnerability Impacts

Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.

Update 1.20.2020 – I'd like to add some additional POSSIBLE impacts, including the possibility that ALL accounts that have gone thru the Citrix Gateway should have their passwords changed right away in addition to other potential impacts. Please review Thomas's article for additional information.

One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you'll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks.
No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don't be that person to use hurtful hashtags or spread fear rather than solutions.

Action Steps for CVE-2019-19781

Last update – 1.17.2020

Fix Script for Citrix NetScaler ADC

Use the instructions at https://support.citrix.com/article/CTX267679 right now. The extended Citrix community is working on additional scripts.
The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down' from the other, you need to replace it with one from your keyboard.
Fortunately – the fix is quick but does require a reboot to take full effect.

UPDATE 1.17.2020

Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I'd do so now.

https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

Check for Vulnerability in your ADC

The US Government has released a method of checking for this flaw. See https://www.us-cert.gov/ncas/current-activity/2020/01/13/cisa-releases-test-citrix-adc-and-gateway-vulnerability and https://github.com/cisagov/check-cve-2019-19781

1.17.2020 – Citrix also released this at https://support.citrix.com/article/CTX269180

Update – 1-14-2020

If someone is cryptomining on your ADC- you'll see high utilization – but there's a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.

Here's what do to. After you've run the prevention script and rebooted, get into the shell, or just enter

shell top -n 10

What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON'T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you're satisfied Ctrl-C will take you out of that.

However, in my mind, cryptomining is a secondary concern. Your company's information may have been exposed at some levels that have not yet been fully determined.

The big indicator of a compromise at this point is .xml files in directories they don't belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you've been exploited, exporting your configuration and configuring from fresh firmware isn't a bad idea.

If you are compromised:

  1. Take the ADC off the network.
  2. Change the password of any LDAP or other AD/network accounts stored on the ADC.
  3. Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
  4. If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
    1. Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
    2. Disconnect the network before starting
    3. Start the machine and verify using the console that the VPX does not appear compromised
    4. Change the nsroot password
    5. Attach the internal network only
    6. Run the fix (alternatively- type this via the console to be safer)
    7. attach the external network
    8. Keep an eye on the logs
  5. Replace SSL Certificates on the appliance at your earliest opportunity

Timeline and Updates

Jan 13 2020
Citrix has announced a timeline for ADC firmware that will include fixes.

Note- these are the initial timelines superseded on Jan 19th

VersionRefresh BuildExpected Release Date
10.510.5.70.x31st January 2020
11.111.1.63.x20th January 2020
12.012.0.63.x20th January 2020
12.112.1.55.x27th January 2020
13.013.0.47.x27th January 2020

Jan 14 2020

I have started working with clients to remediate compromises and double-check other clients. I've updated some suggested quick things above.

I'm tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!

…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.

Jan 16 2020

Citrix released an officially supported way to scan for the vulnerability, though it is not exactly user friendly https://support.citrix.com/article/CTX269180

Jan 17 2020

Citrix Posted this blog article clarifying a few things: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

DJ Eshelman announced that he will be working with a few other industry professionals to create a video and community support course offering to help people remediate this issue.

Jan 19 2020

Citrix blog post with updated firmware releases.

Citrix ADC and Citrix Gateway
VersionRefresh BuildRelease Date
11.111.1.63.15January 19, 2020
12.012.0.63.13January 19, 2020
12.112.1.55.xJanuary 24, 2020
10.510.5.70.xJanuary 24, 2020
13.013.0.47.xJanuary 24, 2020
Citrix SD-WAN WANOP
ReleaseCitrix ADC ReleaseRelease Date
10.2.611.1.51.615January 24, 2020
11.0.311.1.51.615January 24, 2020
Configure Your NetScaler (Citrix ADC) 13 with Carl Stalhood’s new guides

Configure Your NetScaler (Citrix ADC) 13 with Carl Stalhood’s new guides

Carl Stalhood is at it again, with new build guides for Citrix ADC (formerly NetScaler). He has been so helpful over the years so I thought I'd boost the signal to his site a bit. He is a big reason I don't currently make my own guides. Why would I need to? His are great!
We will keep this post up to date as best we can – Carl moves faster than we do, though!

(more…)
NetScaler Security for the XenApp Dummy – Part 2: Design

NetScaler Security for the XenApp Dummy – Part 2: Design

We're back, talking more NetScaler Security! This is part of our four part series using my recommended Assess (or Understand), Design (Plan), Change (Build) and Maintain (Manage) Methodology.

In Part One we went into detail on how to assess and find just how secure your NetScaler configuration really is. We went thru a few of the very many leading practices and how to determine variances from them. If you didn't read that one, go ahead and read it… I'll wait. If you have gone thru the article and have your spreadsheet (or notes), we're ready to continue! Also check back on it from time to time- I'll be updating the article every now and again when new threats or methods emerge!

NetScaler Security Goals

First, let me say outright that we are not taking actions on this step. That will come in Part 3, Change. For now, we are making a plan of action- what do we intend to do in order to address the concerns we have from our Assessment. Setting some goals is key here.

Setting NetScaler Security Goals

Let's go thru the list and determine what we want for the best balance between simplicity to deploy and good NetScaler security. We'll start with the Highest Urgency and Importance items.

  • Score an A+ at SSLLabs.com. Looking from my example, I can see that my NetScaler Gateway is scoring a “C”. I think having an A+ will be a good indicator of acceptable NetScaler security.
  • Ban SSL3 and TLS 1. With TLS 1.2 supported and 1.3 on the horizon, my goal will be to sunset older SSL ciphers completely for better protection.
  • Secure the NetScaler Management Interfaces. I want to make sure that the leading NetScaler security practices are followed to prevent attacks. This will include defining ACLs and restricting access to the NSIP interface. And, of course- change the NetScaler's NSROOT password.
  • Upgrade to the Best Firmware Choice. As I mentioned in Part One, just because a firmware update has been released doesn't mean it is automatically best to use it. In my case, I want to be sure the firmware chosen meets minimums for NetScaler security, not features as my primary concerns.
  • Address Leading Security items. Our assessment indicated at least one leading practice that is security related, so we will validate those settings.

Fine Tune for XenApp and XenDesktop

Several lower priority findings and other items marked as Low Urgency and Importance.

  • Validate Leading Practices. As the Health Check Summary from CIS indicated there were several leading practices in need of adjustment from the defaults. I'll focus on the ones that affect overall NetScaler security.
  • Address Functionality Items. Our assessment had a few items of low priority that may affect the functionality of other systems. In fact, in our case I'm going to even add one that I know needs to be addressed for functionality of the new EDT protocol.

Design

In the Design phase, we will be determining Design Decisions – in this case declaring the configuration changes that will give us the best NetScaler security that makes sense for our use cases.

Score an A+.

I know that to get to A+ I have a lot of things to address, but I am going to first start with a pretty major caveat. Just as I noted in Part 1, we need to declare a minimum support for our SSL settings. If we go too strict, we lose the ability for certain endpoints to connect entirely. On the other hand, we may only want those with secure and updated operating systems to be able to connect at all. For our example, we know that the following is true:

  • We only support connections from up to date browsers and operating systems. All others are best effort and must meet NetScaler security minimums. That means no XP, Vista or out of date OS configurations. Sorry, for those of you stuck in 2008.
  • We do not have Thin clients in scope so the need to continue allowing TLS 1.1 to support older thin clients does not apply. We can also apply STS.

First- let's deal with the same things I mention in my original NetScaler security article about getting the Best SSLLabs Rating (which will be updated from time to time)

First we will go with the most secure Cipher sets and configuration possible. Now- I know a few things going into this that I want to share with you:

  • First- upgrading firmware. In 2018, SSL Labs will begin downgrading scores for those that are vulnerable. When upgrading firmware, I typically start with the minimum I need and go up to the latest firmware as long as it is at least a month old. This is from experience in feature releases causing disruption for other functions. The most common I've seen is problems with AppFlow. If you're using HDX Insight (and let's be honest, you should be) this is something you need to be aware of. If you're not using AppFlow… knock yourself out with the latest; odds are good you'll be fine. But I can't stress enough how important validation testing is when upgrading to new major releases (say, 11 to 12 for example). If you're stuck or less confident- phone a friend. Or contact me using the form over to your right…
  • The way SSL Ciphers are named gets confusing in a hurry- but the names are organized in such a way that indicates a balance between security and performance. NetScaler Gateway has some pretty specific processing needs, especially when using a VPX which doesn't have an offload chip. Therefore, the selections we'll make as primary will be driven by performance first. This means in the case of DHE vs ECDHE, we're going to favor ECHDE because they will perform better in our use case. If you don't know what I'm talking about… that's okay. I barely know myself if I'm being honest. But this what I have tested and deployed to over 50 NetScalers last year. Here's more information if you are hungry for it: https://docs.citrix.com/en-us/netscaler/12/ssl/supported-ciphers-list-release-11.html Note again- that if you are using a FIPS Appliance, you'll want to review this: https://docs.citrix.com/en-us/netscaler/12/ssl/fips-approved-ciphers.html The list gets complicated, but few care more about NetScaler Security than those using FIPS appliances. Best to understand what Ciphers will be supported for your appliance because the mileage varies!
    • We will make a set of Ciphers specific to NetScaler Gateway, which we will refer to as “NSG”
    • We will separate RSA from non-RSA keys for TLS 1.2 and above, because there is drama emerging there right now. To be flexible in the future, we'll make it easy to disable the cipher set for testing.
  • Beware ROBOT. I did a re-scan of my SSLLabs assessment and found that at the end of Feb 2018, my NetScaler security score would be an F! This is due to the firmware being vulnerable to ROBOT attack. You can test for that vulnerability here: https://robotattack.org/. We will update the firmware as part of our process to deal with this threat.
  • Speaking of SSLLabs- another up and coming trend you need to be aware of is Perfect Forward Security (PFS). To explain this as briefly as possible, there are two ways to achieve PFS, by using DHE or ECDHE keys. Based on my limited testing and research, I have found that ECDHE performs better than DHE, though DHE does offer better security. If you are running an MPX or SPX appliance you may want to look into DHE (Diffe-Helman Exchange). I do not yet recommend this for VPX that are not hosted on SPX. A lot more information can be found here: https://support.citrix.com/article/CTX205282 But- as long as the ECC curves are in place (they usually are if you have any ECDHE ciphers bound, which we will), this is not a concern for you until the ‘rules' change to favor DHE more heavily. You may also want to enable STS on your StoreFront servers if you have internal connections- but be aware that this may cause disruption to certain thin clients so TEST TEST and then TEST AGAIN!
  • Note- There are new Ciphers coming soon to support TLS 1.3 – for this article, however I am assuming that we will not have access to that firmware and I'll update this later with those new sets. We'll create a Cipher set name that we will keep updated as the latest and greatest and include “CurrentCiphers” in its name.

Other Recommended Settings

  • Remember that we are going to change our NSROOT Password. I thought about suggesting Active Directory integration for NetScaler as well but that is probably best for another blog post or referring you to other sites. If you'd like to learn more about the process and include it in your document, it is something I recommend as you won't have your NSROOT password floating all over the place. https://support.citrix.com/article/CTX123782 Regardless- change your NSROOT password on a regular basis even if you have AD authentication configured.
  • Unless our testing reveals something different, we plan on implementing all the recommendations found in our Scout report. Rather than type them twice, I'll list them in the design document below.

NetScaler Security Design Document

Pulling from all the items above, we can generate a document that lays out our plan for better NetScaler security so we can share it with others if needs be- but is also very useful if it is something that will take some time to get approved. It doesn't have to be anything fancy, but I always advise people to write things down first – here's something to get you started. PLEASE NOTE- these are just the items I found on my assessment example. You'll want to include any findings you have as well!

ItemDesign DecisionNotes
NSROOT PasswordChange to a complex passwordStore the new password securely! I use Dashlane to store and securely control sharing of passwords with 2FA security. Sign up here to get $20 off for you and give $20 for me too! Way safer than putting them in a file!
NetScaler FirmwareUpgrade Firmware to latest stable within our major version release.We'll be using the 12.0 56.20 build in our example. I chose this simply as it is the latest at the time of writing and met the criteria of not being vulnerable to CVE-2017-14602 and 2017-17382.

Guidance on the steps to expect here:

https://support.citrix.com/article/CTX127455

To prevent ROBOT Attack, make sure to get a firmware above https://support.citrix.com/article/CTX230238

Management InterfacesOnly allow the NSIP as a management interface and force Secure (SSL) communication onlyDisable Enable Management Access control, Telnet, SSH, and GUI on all Non-Management IPs.

More information here about restricting NSIPs to Only allow management applications: https://support.citrix.com/article/CTX126736
See the “Enable Secure Access to NetScaler GUI” at https://support.citrix.com/article/CTX111531

NetScaler Security – Interface ACLs-Allow HA Communication between NetScalers

-Allow communication from Utility servers

-Allow communication from NMAS Server

-Deny all others

This action must be performed on both NetScalers

NetScaler NSIPs: (list yours)
Utility Server IPs: (list servers or subnets that will have access)
NetScaler MAS: (If you have them- list the IP Addresses of NetScaler MAS appliances)
NetScaler Gateway – SSL ParametersTLS 1.2 only
Enable HSTS (Strict Transport Security
Disallow SSL v3, TLS 1 and 1.1

For HSTS, configure a rewrite action.

Read up on it at https://support.citrix.com/article/CTX205221

Note: this procedure is different depending on if you are using a 12.x based firmware or not. In our case we can use the instructions for 12.x at https://support.citrix.com/article/CTX224172

NetScaler Gateway – Basic SettingsEnable DTLSCheck box for DTLS
Traffic Management – SSL – Cipher GroupsCreate custom Cipher GroupsNSG-TLS1.2-RSA-Ciphers

NSG-LegacyCiphers

TLS-HighSecureCurrentCiphers

Traffic Management – SSL – CiphersFavor high performance, High security ciphers and work downward within each set.

TLS1.2-RSA group will be isolated for future removal if needed.

NSG-LegacyCiphers will only be used if older clients are in place, but I recommend creating the group just in case.

Each grouping will start with higher AES and SHA values and work downward.

TLS-HighSecureCurrentCiphers group will utilize the new ECHDE-ECDSA ciphers instead of RSA ciphers.

NSG-TLS1.2-RSA-Ciphers:
TLS1.2-ECDHE-RSA-AES256-GCM-SHA384TLS1.2-ECDHE-RSA-AES128-GCM-SHA256TLS1.2-ECDHE-RSA-AES-256-SHA384TLS1.2-ECDHE-RSA-AES-128-SHA256TLS1.2-DHE-RSA-AES256-GCM-SHA384TLS1.2-DHE-RSA-AES128-GCM-SHA256NSG-LegacyCiphers:

TLS1-ECDHE-RSA-AES256-SHA

TLS1-ECDHE-RSA-AES128-SHA

TLS1-DHE-RSA-AES-256-CBC-SHA

TLS1-DHE-RSA-AES-128-CBC-SHA

TLS1-AES-256-CBC-SHA

TLS1-AES-128-CBC-SHA

TLS-HighSecureCurrentCiphers:

TLS1.2-ECDHE-ECDSA-AES256-SHA384

TLS1.2-ECDHE-ECDSA-AES128-SHA256

TLS1.2-ECDHE-ECDSA-AES256-GCM-SHA384

TLS1.2-ECDHE-ECDSA-AES128-GCM-SHA256

SSL ParametersDisable Client and Server Side SSL RenegotiationDeny SSL Regeneration to the Frontend Client

https://support.citrix.com/article/CTX123680

TCP DefaultsEnable Nagle's Algorithm

Enable Selective Acknowledgement

Enable Window Scaling

https://support.citrix.com/article/CTX121149

Note- check with your Network Administrator to be sure the Window Scaling option will be supported (SACK is tied to Scaling.) If not, only list Nagle's. Another key metric here is the Factor for Scaling. Typically 4 is correct but see the article at https://support.citrix.com/article/CTX113656 to be sure you will not need to adjust this value after conferring with your Network Administrator.

HTTP ParametersDrop invalid HTTP requests

Deployment Plan

Whenever possible, I believe in using a non-production test environment or components to validate changes before impacting users. Part of our Deployment Plan will involve testing these settings on a test NetScaler VPX prior to deployment in production. Once our changes are validated, we'll set a time for a production outage so that we can roll back changes if required. For an HA deployment, it is very possible to do the updates in a way that minimizes production impacts. However, with NetScaler Gateway connections will be disrupted when we reboot. So whenever possible with this kind of update, I recommend declaring an outage or at the least setting the expectation that sessions may disrupted during a defined time window when the changes will be made.

So write out a deployment plan with a rollback plan. This is very useful especially when you must generate something for a Change Control board. Here's the high-level plan:

  1. Save the Running Configuration of the test NetScaler VPX
  2. Export the full NetScaler Backup I recommend reviewing my fellow CTA George Spiers' guide at http://www.jgspiers.com/netscaler-backup-restore/
  3. Run a snapshot of the NetScaler VPX
  4. Upgrade NetScaler Firmware (if HA, use the advice in the article above to list out your steps)
  5. Secure the NSIP Interface by introducing new ACLs (VERIFY before you continue)
  6. Make the other changes noted above

Next time- I'll guide you thru the process of actually making the changes! When we're done, you'll have NetScaler Security at a level of confidence that will exceed most enterprise customers I've been visiting lately!

Citrix Monitoring Webinar with ControlUP on 11/29/2017 Update: Recording now available

Citrix Monitoring Webinar with ControlUP on 11/29/2017 Update: Recording now available

(Note: This article has been updated 12/1/2017)

The worst time to find out about problems in your Citrix environment is after they are already happening. But the built-in tools (Director) don't really always paint the full picture to give you quality Citrix monitoring. You need Proactive Citrix Monitoring and ControlUp is here to help!

Citrix Monitoring Webinar on 11-29-2017

Yoni Avital (Founder & CTO of ControlUp) and I had an hour long CUGC Connect webinar hosted by the Citrix User Group Community! Wednesday, November 29th 2017 at noon Central (US).

Yoni first showed me ControlUp almost 4 years ago and I've been enjoying watching them improve every year. I have loved the single-pane-of-glass approach they take to environment monitoring and engagement.

Why View the Replay?

Aside from hearing my soothing voice moderate… Yoni demoed ControlUp 7.1 which I believe excels not only at Citrix monitoring (as in XenApp/XenDesktop) but NetScaler monitoring as well!

New features also include adding nVidia vGPU at the VM and process level to your Citrix monitoring, metrics of your published applications (yay!) and a new troubleshooting feature.

 

Will there be a Demo?

YES- you'll be able to see this all in action live in the webinar.

Is it Free?

YES- but you will need to sign up for the community at myCUGC.org and then view the replay at https://www.mycugc.org/p/fo/st/thread=2356

There were a lot of questions that were asked during the webinar- an unprecidented amount from the over 250 people attending!

So we gave Yoni all the questions that weren't answered and he wrote them out in the thread!

I would love your feedback on the webinar, too.  Leave me a comment or connect with me on Twitter (@TheXenMaster)

As always- be sure to sign up for my free newsletter to always be informed of cool happenings like this!

A few notes:

  • This is not a paid endorsement of any product nor am I receiving any compensation for this webinar
  • I am a leader at CUGC and promote because I love it
  • I am moderating this webinar as a proud member of the Citrix Technology Advocates program
NetScaler Security for the XenApp Dummy – Part 1: Assess

NetScaler Security for the XenApp Dummy – Part 1: Assess

So you have this “NetScaler” thing to front end your XenApp or XenDesktop environment… But maybe you are like me and NetScaler Security isn't what you spend most of your day dealing with. So, how can you make sure in light of recent security threats that it is running properly? In a post in 2016 I discussed how to get an A+ Rating at SSL Labs for your NetScaler Gateway in under 5 minutes. I figured it was time for an update for 2017 taking some new things into consideration but approach this from the point of view of someone like me that isn't “A NetScaler Person.”

(last update: January, 2019)

(more…)

Categories

By With a Little Help from Our Friends

ByteSized Book logo