Writing a book is… really hard. But what if you could easily help one be written?
Well, that is exactly the opportunity in front of you!
Instead of writing chapters, editing, more writing, more editing and then distribution, worry, stress and…
okay, I may be relating some personal experience here…
As I mentioned in my February Newsletter – You can participate in a unique project… by submitting just 250 words.
This is the collaborative effort of Christiaan Brinkhoff and Bas van Kaam – the Byte-Sized Book Project.
The question is – do you have something you'd like to say about Cloud design principles, leading practices or even recommended reference builds you'd like to relay to the world? This is your shot to be heard!
This is all about creating value for the community as a whole, but not asking for a ton from each person. I think it's a brilliant approach!
So I encourage you to go for it! But don't wait- the plan is to get the book edited and done in the next few months!
THE IDEA BEHIND THIS PROJECT IS SIMPLE, WE ARE LOOKING FOR AS MANY CLOUD DESIGN PRINCIPLES, BEST OR COMMON PRACTICES, QUOTES, AND ARCHITECTURAL RECOMMENDATIONS AS POSSIBLE. FOR THE COMMUNITY, BY THE COMMUNITY!
The goal of this series is to outline some of the more common Citrix Mistakes that I have been seeing in my consulting engagements. These top three were chosen not because of frequency per se, but for the sheer impact they have. I chose them so that when you implement the fixes – you can be the hero. The #CitrixHero.
We're back, talking more NetScaler Security! This is part of our four part series using my recommended Assess (or Understand), Design (Plan), Change (Build) and Maintain (Manage) Methodology.
In Part One we went into detail on how to assess and find just how secure your NetScaler configuration really is. We went thru a few of the very many leading practices and how to determine variances from them. If you didn't read that one, go ahead and read it… I'll wait. If you have gone thru the article and have your spreadsheet (or notes), we're ready to continue! Also check back on it from time to time- I'll be updating the article every now and again when new threats or methods emerge!
NetScaler Security Goals
First, let me say outright that we are not taking actions on this step. That will come in Part 3, Change. For now, we are making a plan of action- what do we intend to do in order to address the concerns we have from our Assessment. Setting some goals is key here.
Setting NetScaler Security Goals
Let's go thru the list and determine what we want for the best balance between simplicity to deploy and good NetScaler security. We'll start with the Highest Urgency and Importance items.
Score an A+ at SSLLabs.com. Looking from my example, I can see that my NetScaler Gateway is scoring a “C”. I think having an A+ will be a good indicator of acceptable NetScaler security.
Ban SSL3 and TLS 1. With TLS 1.2 supported and 1.3 on the horizon, my goal will be to sunset older SSL ciphers completely for better protection.
Secure the NetScaler Management Interfaces. I want to make sure that the leading NetScaler security practices are followed to prevent attacks. This will include defining ACLs and restricting access to the NSIP interface. And, of course- change the NetScaler's NSROOT password.
Upgrade to the Best Firmware Choice. As I mentioned in Part One, just because a firmware update has been released doesn't mean it is automatically best to use it. In my case, I want to be sure the firmware chosen meets minimums for NetScaler security, not features as my primary concerns.
Address Leading Security items. Our assessment indicated at least one leading practice that is security related, so we will validate those settings.
Fine Tune for XenApp and XenDesktop
Several lower priority findings and other items marked as Low Urgency and Importance.
Validate Leading Practices. As the Health Check Summary from CIS indicated there were several leading practices in need of adjustment from the defaults. I'll focus on the ones that affect overall NetScaler security.
Address Functionality Items. Our assessment had a few items of low priority that may affect the functionality of other systems. In fact, in our case I'm going to even add one that I know needs to be addressed for functionality of the new EDT protocol.
In the Design phase, we will be determining Design Decisions – in this case declaring the configuration changes that will give us the best NetScaler security that makes sense for our use cases.
Score an A+.
I know that to get to A+ I have a lot of things to address, but I am going to first start with a pretty major caveat. Just as I noted in Part 1, we need to declare a minimum support for our SSL settings. If we go too strict, we lose the ability for certain endpoints to connect entirely. On the other hand, we may only want those with secure and updated operating systems to be able to connect at all. For our example, we know that the following is true:
We only support connections from up to date browsers and operating systems. All others are best effort and must meet NetScaler security minimums. That means no XP, Vista or out of date OS configurations. Sorry, for those of you stuck in 2008.
We do not have Thin clients in scope so the need to continue allowing TLS 1.1 to support older thin clients does not apply. We can also apply STS.
First- let's deal with the same things I mention in my original NetScaler security article about getting the Best SSLLabs Rating (which will be updated from time to time)
First we will go with the most secure Cipher sets and configuration possible. Now- I know a few things going into this that I want to share with you:
First- upgrading firmware. In 2018, SSL Labs will begin downgrading scores for those that are vulnerable. When upgrading firmware, I typically start with the minimum I need and go up to the latest firmware as long as it is at least a month old. This is from experience in feature releases causing disruption for other functions. The most common I've seen is problems with AppFlow. If you're using HDX Insight (and let's be honest, you should be) this is something you need to be aware of. If you're not using AppFlow… knock yourself out with the latest; odds are good you'll be fine. But I can't stress enough how important validation testing is when upgrading to new major releases (say, 11 to 12 for example). If you're stuck or less confident- phone a friend. Or contact me using the form over to your right…
The way SSL Ciphers are named gets confusing in a hurry- but the names are organized in such a way that indicates a balance between security and performance. NetScaler Gateway has some pretty specific processing needs, especially when using a VPX which doesn't have an offload chip. Therefore, the selections we'll make as primary will be driven by performance first. This means in the case of DHE vs ECDHE, we're going to favor ECHDE because they will perform better in our use case. If you don't know what I'm talking about… that's okay. I barely know myself if I'm being honest. But this what I have tested and deployed to over 50 NetScalers last year. Here's more information if you are hungry for it: https://docs.citrix.com/en-us/netscaler/12/ssl/supported-ciphers-list-release-11.html Note again- that if you are using a FIPS Appliance, you'll want to review this: https://docs.citrix.com/en-us/netscaler/12/ssl/fips-approved-ciphers.html The list gets complicated, but few care more about NetScaler Security than those using FIPS appliances. Best to understand what Ciphers will be supported for your appliance because the mileage varies!
We will make a set of Ciphers specific to NetScaler Gateway, which we will refer to as “NSG”
We will separate RSA from non-RSA keys for TLS 1.2 and above, because there is drama emerging there right now. To be flexible in the future, we'll make it easy to disable the cipher set for testing.
Beware ROBOT. I did a re-scan of my SSLLabs assessment and found that at the end of Feb 2018, my NetScaler security score would be an F! This is due to the firmware being vulnerable to ROBOT attack. You can test for that vulnerability here: https://robotattack.org/. We will update the firmware as part of our process to deal with this threat.
Speaking of SSLLabs- another up and coming trend you need to be aware of is Perfect Forward Security (PFS). To explain this as briefly as possible, there are two ways to achieve PFS, by using DHE or ECDHE keys. Based on my limited testing and research, I have found that ECDHE performs better than DHE, though DHE does offer better security. If you are running an MPX or SPX appliance you may want to look into DHE (Diffe-Helman Exchange). I do not yet recommend this for VPX that are not hosted on SPX. A lot more information can be found here: https://support.citrix.com/article/CTX205282 But- as long as the ECC curves are in place (they usually are if you have any ECDHE ciphers bound, which we will), this is not a concern for you until the ‘rules' change to favor DHE more heavily. You may also want to enable STS on your StoreFront servers if you have internal connections- but be aware that this may cause disruption to certain thin clients so TEST TEST and then TEST AGAIN!
Note- There are new Ciphers coming soon to support TLS 1.3 – for this article, however I am assuming that we will not have access to that firmware and I'll update this later with those new sets. We'll create a Cipher set name that we will keep updated as the latest and greatest and include “CurrentCiphers” in its name.
Other Recommended Settings
Remember that we are going to change our NSROOT Password. I thought about suggesting Active Directory integration for NetScaler as well but that is probably best for another blog post or referring you to other sites. If you'd like to learn more about the process and include it in your document, it is something I recommend as you won't have your NSROOT password floating all over the place. https://support.citrix.com/article/CTX123782 Regardless- change your NSROOT password on a regular basis even if you have AD authentication configured.
Unless our testing reveals something different, we plan on implementing all the recommendations found in our Scout report. Rather than type them twice, I'll list them in the design document below.
NetScaler Security Design Document
Pulling from all the items above, we can generate a document that lays out our plan for better NetScaler security so we can share it with others if needs be- but is also very useful if it is something that will take some time to get approved. It doesn't have to be anything fancy, but I always advise people to write things down first – here's something to get you started. PLEASE NOTE- these are just the items I found on my assessment example. You'll want to include any findings you have as well!
Change to a complex password
Store the new password securely! I use Dashlane to store and securely control sharing of passwords with 2FA security. Sign up here to get $20 off for you and give $20 for me too! Way safer than putting them in a file!
Upgrade Firmware to latest stable within our major version release.
We'll be using the 12.0 56.20 build in our example. I chose this simply as it is the latest at the time of writing and met the criteria of not being vulnerable to CVE-2017-14602 and 2017-17382.
Note- check with your Network Administrator to be sure the Window Scaling option will be supported (SACK is tied to Scaling.) If not, only list Nagle's. Another key metric here is the Factor for Scaling. Typically 4 is correct but see the article at https://support.citrix.com/article/CTX113656 to be sure you will not need to adjust this value after conferring with your Network Administrator.
Drop invalid HTTP requests
Whenever possible, I believe in using a non-production test environment or components to validate changes before impacting users. Part of our Deployment Plan will involve testing these settings on a test NetScaler VPX prior to deployment in production. Once our changes are validated, we'll set a time for a production outage so that we can roll back changes if required. For an HA deployment, it is very possible to do the updates in a way that minimizes production impacts. However, with NetScaler Gateway connections will be disrupted when we reboot. So whenever possible with this kind of update, I recommend declaring an outage or at the least setting the expectation that sessions may disrupted during a defined time window when the changes will be made.
So write out a deployment plan with a rollback plan. This is very useful especially when you must generate something for a Change Control board. Here's the high-level plan:
Save the Running Configuration of the test NetScaler VPX
Upgrade NetScaler Firmware (if HA, use the advice in the article above to list out your steps)
Secure the NSIP Interface by introducing new ACLs (VERIFY before you continue)
Make the other changes noted above
Next time- I'll guide you thru the process of actually making the changes! When we're done, you'll have NetScaler Security at a level of confidence that will exceed most enterprise customers I've been visiting lately!
Gotta boost the signal on this. My friend Nick Rintalan from Citrix Consulting has put together a new ‘best practice' (or leading practices for the lawyers) update that I feel it's important for people to see!
Nick Rintalan, Lead Architect at Citrix Consulting
New Best Practice(s)?
Here are some of the highlights of the article, sorted here by what I feel is most important for you to read:
PVS and Memory Buffers. Yes, yes, for the love of all that is holy, yes. I haven't yet deployed for validated the Write Cache features now in MCS, but I can tell you from experience that XenApp with 2-4 GB of RAM cache with failover to disk has been giving roughly 20-30% faster logons and overall better experience for most of my customers.
Protocols (as in HDX). One of my primary frustrations for quite some time now is that Citrix XenDesktop ships by default with a protocol that has a good experience on LAN but tends to be problematic at distance. H.264 is great for video, but frankly I hate it everywhere else. I think it almost singlehandedly ruined things for Citrix since PCoIP can perform better than this hog (my opinion). Thinwire and even the legacy encoder, however- actually deliver on the promises and need to be investigated in nearly every single use case I see. So I agree with Nick- use the policy templates included with 7.6 u3 and above (including LTSR) as a starting point. Odds are good you won't be disappointed. When I say ‘use' here what I mean is remember that you can apply these codecs on a per user basis, connection basis or even per delivery group- meaning filters are your friend! It is perfectly acceptable to have multiple codecs going for various use cases. One size nearly NEVER fits all, so test these out!
vSphere Cluster Sizing. Number 3 on my list right now. You need a dedicated resource cluster for Enterprise workloads- but honestly- for XenApp workloads, consider more hosts per cluster. You should be using bigger VMs anyway, so the number of managed VMs is about the same- just more computing power. CCS is seeing 24+ hosts per cluster be just fine in XenApp. For XenDesktop with more than 5000 VMs- I will add here that a dedicated vCenter may save you a lot of pain… my opinion, and of course… you guessed it. TEST!
XenApp CPU Over-Subscription. Seriously. The “1.5x” thing needed an update so I'm glad to see some clarification here. In all things- I still encourage practical testing instead of just implementing something because “Citrix said to.”
PVS Ports and Threads. Those of you who know me know I bang this drum a lot- so here's some backup for what I'm saying. The defaults are not good enough. Good design is still required!
Farm Design. You're probably like me and are coming along kicking and screaming from XenApp 6.5, which most would agree has been the “Windows XP” of the Citrix world. It just hasn't been this good yet, and I still feel 7.9 doesn't have true feature parity… but as Nick describes, they are getting there. As always… TEST, TEST and then TEST some more before you implement zones with FMA!!!!
XenMobile Optimizations. I guess we have to talk about it. XenMobile is here to stay, so best to not take the ‘out of the box' experience there either.