We’re back, talking more NetScaler Security! This is part of our four part series using my recommended Assess (or Understand), Design (Plan), Change (Build) and Maintain (Manage) Methodology.
In Part One we went into detail on how to assess and find just how secure your NetScaler configuration really is. We went thru a few of the very many leading practices and how to determine variances from them. If you didn’t read that one, go ahead and read it… I’ll wait. If you have gone thru the article and have your spreadsheet (or notes), we’re ready to continue! Also check back on it from time to time- I’ll be updating the article every now and again when new threats or methods emerge!
NetScaler Security Goals
First, let me say outright that we are not taking actions on this step. That will come in Part 3, Change. For now, we are making a plan of action- what do we intend to do in order to address the concerns we have from our Assessment. Setting some goals is key here.
Setting NetScaler Security Goals
Let’s go thru the list and determine what we want for the best balance between simplicity to deploy and good NetScaler security. We’ll start with the Highest Urgency and Importance items.
- Score an A+ at SSLLabs.com. Looking from my example, I can see that my NetScaler Gateway is scoring a “C”. I think having an A+ will be a good indicator of acceptable NetScaler security.
- Ban SSL3 and TLS 1. With TLS 1.2 supported and 1.3 on the horizon, my goal will be to sunset older SSL ciphers completely for better protection.
- Secure the NetScaler Management Interfaces. I want to make sure that the leading NetScaler security practices are followed to prevent attacks. This will include defining ACLs and restricting access to the NSIP interface. And, of course- change the NetScaler’s NSROOT password.
- Upgrade to the Best Firmware Choice. As I mentioned in Part One, just because a firmware update has been released doesn’t mean it is automatically best to use it. In my case, I want to be sure the firmware chosen meets minimums for NetScaler security, not features as my primary concerns.
- Address Leading Security items. Our assessment indicated at least one leading practice that is security related, so we will validate those settings.
Fine Tune for XenApp and XenDesktop
Several lower priority findings and other items marked as Low Urgency and Importance.
- Validate Leading Practices. As the Health Check Summary from CIS indicated there were several leading practices in need of adjustment from the defaults. I’ll focus on the ones that affect overall NetScaler security.
- Address Functionality Items. Our assessment had a few items of low priority that may affect the functionality of other systems. In fact, in our case I’m going to even add one that I know needs to be addressed for functionality of the new EDT protocol.
In the Design phase, we will be determining Design Decisions – in this case declaring the configuration changes that will give us the best NetScaler security that makes sense for our use cases.
Score an A+.
I know that to get to A+ I have a lot of things to address, but I am going to first start with a pretty major caveat. Just as I noted in Part 1, we need to declare a minimum support for our SSL settings. If we go too strict, we lose the ability for certain endpoints to connect entirely. On the other hand, we may only want those with secure and updated operating systems to be able to connect at all. For our example, we know that the following is true:
- We only support connections from up to date browsers and operating systems. All others are best effort and must meet NetScaler security minimums. That means no XP, Vista or out of date OS configurations. Sorry, for those of you stuck in 2008.
- We do not have Thin clients in scope so the need to continue allowing TLS 1.1 to support older thin clients does not apply. We can also apply STS.
First- let’s deal with the same things I mention in my original NetScaler security article about getting the Best SSLLabs Rating (which will be updated from time to time)
First we will go with the most secure Cipher sets and configuration possible. Now- I know a few things going into this that I want to share with you:
- First- upgrading firmware. In 2018, SSL Labs will begin downgrading scores for those that are vulnerable. When upgrading firmware, I typically start with the minimum I need and go up to the latest firmware as long as it is at least a month old. This is from experience in feature releases causing disruption for other functions. The most common I’ve seen is problems with AppFlow. If you’re using HDX Insight (and let’s be honest, you should be) this is something you need to be aware of. If you’re not using AppFlow… knock yourself out with the latest; odds are good you’ll be fine. But I can’t stress enough how important validation testing is when upgrading to new major releases (say, 11 to 12 for example). If you’re stuck or less confident- phone a friend. Or contact me using the form over to your right…
The way SSL Ciphers are named gets confusing in a hurry- but the names are organized in such a way that indicates a balance between security and performance. NetScaler Gateway has some pretty specific processing needs, especially when using a VPX which doesn’t have an offload chip. Therefore, the selections we’ll make as primary will be driven by performance first. This means in the case of DHE vs ECDHE, we’re going to favor ECHDE because they will perform better in our use case. If you don’t know what I’m talking about… that’s okay. I barely know myself if I’m being honest. But this what I have tested and deployed to over 50 NetScalers last year. Here’s more information if you are hungry for it: https://docs.citrix.com/en-us/netscaler/12/ssl/supported-ciphers-list-release-11.html Note again- that if you are using a FIPS Appliance, you’ll want to review this: https://docs.citrix.com/en-us/netscaler/12/ssl/fips-approved-ciphers.html The list gets complicated, but few care more about NetScaler Security than those using FIPS appliances. Best to understand what Ciphers will be supported for your appliance because the mileage varies!
- We will make a set of Ciphers specific to NetScaler Gateway, which we will refer to as “NSG”
- We will separate RSA from non-RSA keys for TLS 1.2 and above, because there is drama emerging there right now. To be flexible in the future, we’ll make it easy to disable the cipher set for testing.
- Beware ROBOT. I did a re-scan of my SSLLabs assessment and found that at the end of Feb 2018, my NetScaler security score would be an F! This is due to the firmware being vulnerable to ROBOT attack. You can test for that vulnerability here: https://robotattack.org/. We will update the firmware as part of our process to deal with this threat.
- Speaking of SSLLabs- another up and coming trend you need to be aware of is Perfect Forward Security (PFS). To explain this as briefly as possible, there are two ways to achieve PFS, by using DHE or ECDHE keys. Based on my limited testing and research, I have found that ECDHE performs better than DHE, though DHE does offer better security. If you are running an MPX or SPX appliance you may want to look into DHE (Diffe-Helman Exchange). I do not yet recommend this for VPX that are not hosted on SPX. A lot more information can be found here: https://support.citrix.com/article/CTX205282 But- as long as the ECC curves are in place (they usually are if you have any ECDHE ciphers bound, which we will), this is not a concern for you until the ‘rules’ change to favor DHE more heavily. You may also want to enable STS on your StoreFront servers if you have internal connections- but be aware that this may cause disruption to certain thin clients so TEST TEST and then TEST AGAIN!
- Note- There are new Ciphers coming soon to support TLS 1.3 – for this article, however I am assuming that we will not have access to that firmware and I’ll update this later with those new sets. We’ll create a Cipher set name that we will keep updated as the latest and greatest and include “CurrentCiphers” in its name.
Other Recommended Settings
- Remember that we are going to change our NSROOT Password. I thought about suggesting Active Directory integration for NetScaler as well but that is probably best for another blog post or referring you to other sites. If you’d like to learn more about the process and include it in your document, it is something I recommend as you won’t have your NSROOT password floating all over the place. https://support.citrix.com/article/CTX123782 Regardless- change your NSROOT password on a regular basis even if you have AD authentication configured.
- Unless our testing reveals something different, we plan on implementing all the recommendations found in our Scout report. Rather than type them twice, I’ll list them in the design document below.
NetScaler Security Design Document
Pulling from all the items above, we can generate a document that lays out our plan for better NetScaler security so we can share it with others if needs be- but is also very useful if it is something that will take some time to get approved. It doesn’t have to be anything fancy, but I always advise people to write things down first – here’s something to get you started. PLEASE NOTE- these are just the items I found on my assessment example. You’ll want to include any findings you have as well!
|NSROOT Password||Change to a complex password||Store the new password securely! I use Dashlane to store and securely control sharing of passwords with 2FA security. Sign up here to get $20 off for you and give $20 for me too! Way safer than putting them in a file!|
|NetScaler Firmware||Upgrade Firmware to latest stable within our major version release.||We’ll be using the 12.0 56.20 build in our example. I chose this simply as it is the latest at the time of writing and met the criteria of not being vulnerable to CVE-2017-14602 and 2017-17382.
Guidance on the steps to expect here:
To prevent ROBOT Attack, make sure to get a firmware above https://support.citrix.com/article/CTX230238
|Management Interfaces||Only allow the NSIP as a management interface and force Secure (SSL) communication only||Disable Enable Management Access control, Telnet, SSH, and GUI on all Non-Management IPs.
More information here about restricting NSIPs to Only allow management applications: https://support.citrix.com/article/CTX126736
|NetScaler Security – Interface ACLs||-Allow HA Communication between NetScalers
-Allow communication from Utility servers
-Allow communication from NMAS Server
-Deny all others
This action must be performed on both NetScalers
|NetScaler Gateway – SSL Parameters||TLS 1.2 only
Enable HSTS (Strict Transport Security
|Disallow SSL v3, TLS 1 and 1.1
For HSTS, configure a rewrite action.
Read up on it at https://support.citrix.com/article/CTX205221
Note: this procedure is different depending on if you are using a 12.x based firmware or not. In our case we can use the instructions for 12.x at https://support.citrix.com/article/CTX224172
|NetScaler Gateway – Basic Settings||Enable DTLS||Check box for DTLS|
|Traffic Management – SSL – Cipher Groups||Create custom Cipher Groups||NSG-TLS1.2-RSA-Ciphers
|Traffic Management – SSL – Ciphers||Favor high performance, High security ciphers and work downward within each set.
TLS1.2-RSA group will be isolated for future removal if needed.
NSG-LegacyCiphers will only be used if older clients are in place, but I recommend creating the group just in case.
Each grouping will start with higher AES and SHA values and work downward.
TLS-HighSecureCurrentCiphers group will utilize the new ECHDE-ECDSA ciphers instead of RSA ciphers.
|SSL Parameters||Disable Client and Server Side SSL Renegotiation||Deny SSL Regeneration to the Frontend Client|
|TCP Defaults||Enable Nagle’s Algorithm
Enable Selective Acknowledgement
Enable Window Scaling
Note- check with your Network Administrator to be sure the Window Scaling option will be supported (SACK is tied to Scaling.) If not, only list Nagle’s. Another key metric here is the Factor for Scaling. Typically 4 is correct but see the article at https://support.citrix.com/article/CTX113656 to be sure you will not need to adjust this value after conferring with your Network Administrator.
|HTTP Parameters||Drop invalid HTTP requests|
Whenever possible, I believe in using a non-production test environment or components to validate changes before impacting users. Part of our Deployment Plan will involve testing these settings on a test NetScaler VPX prior to deployment in production. Once our changes are validated, we’ll set a time for a production outage so that we can roll back changes if required. For an HA deployment, it is very possible to do the updates in a way that minimizes production impacts. However, with NetScaler Gateway connections will be disrupted when we reboot. So whenever possible with this kind of update, I recommend declaring an outage or at the least setting the expectation that sessions may disrupted during a defined time window when the changes will be made.
So write out a deployment plan with a rollback plan. This is very useful especially when you must generate something for a Change Control board. Here’s the high-level plan:
- Save the Running Configuration of the test NetScaler VPX
- Export the full NetScaler Backup I recommend reviewing my fellow CTA George Spiers’ guide at http://www.jgspiers.com/netscaler-backup-restore/
- Run a snapshot of the NetScaler VPX
- Upgrade NetScaler Firmware (if HA, use the advice in the article above to list out your steps)
- Secure the NSIP Interface by introducing new ACLs (VERIFY before you continue)
- Make the other changes noted above
Next time- I’ll guide you thru the process of actually making the changes! When we’re done, you’ll have NetScaler Security at a level of confidence that will exceed most enterprise customers I’ve been visiting lately!