Writing a book is… really hard. But what if you could easily help one be written?
Well, that is exactly the opportunity in front of you!
Instead of writing chapters, editing, more writing, more editing and then distribution, worry, stress and…
okay, I may be relating some personal experience here…
As I mentioned in my February Newsletter – You can participate in a unique project… by submitting just 250 words.
This is the collaborative effort of Christiaan Brinkhoff and Bas van Kaam – the Byte-Sized Book Project.
The question is – do you have something you’d like to say about Cloud design principles, leading practices or even recommended reference builds you’d like to relay to the world? This is your shot to be heard!
This is all about creating value for the community as a whole, but not asking for a ton from each person. I think it’s a brilliant approach!
So I encourage you to go for it! But don’t wait- the plan is to get the book edited and done in the next few months!
THE IDEA BEHIND THIS PROJECT IS SIMPLE, WE ARE LOOKING FOR AS MANY CLOUD DESIGN PRINCIPLES, BEST OR COMMON PRACTICES, QUOTES, AND ARCHITECTURAL RECOMMENDATIONS AS POSSIBLE. FOR THE COMMUNITY, BY THE COMMUNITY!
The goal of this series is to outline some of the more common Citrix Mistakes that I have been seeing in my consulting engagements. These top three were chosen not because of frequency per se, but for the sheer impact they have. I chose them so that when you implement the fixes – you can be the hero. The #CitrixHero.
In the name of security, Google may have made Chrome an even larger burden for virtual workspaces. The change to the latest version of the browser will start rendering each page in its own memory and process space. This is good for security (think Spectre) … But a nightmare for virtual workspaces, especially Server OS VDA (XenApp).
Unfortunately, Citrix has done it again with Citrix Workspace… showing off something pretty cool and adeptly showing off how it works… but not really explaining very well at Synergy how it will be deployed. So I thought I would get some down-to-Earth thoughts together of what’s going on up in the clouds from an Architect perspective.
Feeling lazy or just don’t like reading? The long and short of it is that you can aggregate Cloud Control and On-Prem Control for your resources by purchasing or upgrading to the Citrix Workspace Service. Still confused? I thought you may be. Maybe it’s time to take a few minutes and read this one… But before you do here’s an important methodology lesson:
User/Subscriber Layer – This is the users themselves and any peripherals they attach to. It defines how users use a product.
Access Layer – This is how access to applications is controlled. In the Citrix world we’re largely talking StoreFront, NetScaler Gateway and if you are still living in 2003, VPN.
Resource Layer – This is important! The Resource layer is the… you guessed it, resources the users need access to. Be it Applications hosted on Server OS, VDI Desktops or even SaaS apps. The Resource Layer defines what they are, where they are located and most importantly- how they are structured and maintained.
Control Layer – This is what we define in the Citrix world as the components that aggregate and control resources. It includes Active Directory, Databases, Licensing and services such as the Delivery Controller. An important element of the Control layer is policies.
Cloud (Hardware) Layer – All these services have to live somewhere, and this layer defines it.
Operations Layer – Someone has to maintain all of this- and the Operations Layer is where we define all of this.
First off- when we are talking Workspace SERVICE, we are talking about the ‘everything’ subscription. This means it is Apps and Desktops, Enterprise Mobility Management (MDM/MAM), File Sync and Sharing (ShareFile), and Networking (Which now includes both MAS and the Web App Security Service)… but also the Secure Browser service which is NOT included with the XenApp and XenDesktop service. You can find the full list of the checkboxes here. Your mileage may vary of course, but if you are using even two of the ‘core’ services, Workspace makes sense.
Citrix Pricing for basic cloud services as of June 11, 2018 (source: Citrix.com)
At an estimated $34.38 per user per month (Beginning prices as of June 11, 2018), Workspace is certainly not cheap… but as far as powering productivity from any location… I have to admit, this is THE premium service to make it happen these days.
So what is it giving you? Essentially, we are Hybridizing elements of the Access, Control and Operations Layer. You are still ultimately responsible for maintaining your Resources, regardless of if their location!! This is an important part that is VERY often missed when talking about Citrix Cloud, so please don’t be one of those confused by it! In other words- you still maintain your own Cloud Layer. Citrix doesn’t magically start giving you VDI from their cloud. The exception to this is Secure Browser… but that’s a different topic.
The Citrix Workspace App is the new way of aggregating resources FROM THE WORKSPACE SERVICES. It makes a lot of sense because you can aggregate multiple resources, control mobile enrollment and file management all in one- very nice. But this of course brought up a lot of other questions, which largely are answered here. Note that what was shown at Synergy does require the Workspace Service Premium edition. If you are looking at this and thinking there’s something familiar about it… you’re right. The concept comes from XenMobile’s Secure Hub- but is now brought to other devices as well as mobile. It will be generally available in Q3, with additional features coming in terms of what will happen with the ShareFile Service (that’s a whole other topic).
IMPORTANT- Workspace App WILL REPLACE Receiver sometime in Q3!
Workspace App is also important because it has significant areas of impact in the User and Access layers.
Workspace Service Can Aggregate Cloud and On-Premises Control
Up until recently it was true that if you have an on-premises (or on-prem, but never on-premise, please) Site or sites already set up, that you had to migrate to Cloud control for your resources if you bought that service. This is no longer true because of Site Aggregation. You’ll be able to run both on-prem services (including local NetScaler Gateway and StoreFront) alongside the Workspace App from the cloud!
Danny Feller explains in this video blog that the Workspace service aggregates between both on-prem XenDesktop and Cloud XenApp & XenDesktop Service control planes, using the same unified Workspace App. This capability does NOT exist for existing Cloud XenApp & XenDesktop Service customers, however.
Why This Matters
To date this has been an enormous barrier to adoption, especially in places where the skillsets already exist to maintain local resources. One thing that has not been talked about yet is how this impacts Multi-Site deployments: If you can install a connector and the account you’re using for logon is valid or federated… you should be able to aggregate multiple resources thru the Workspace App. This is significant for a lot of companies dealing with M&A because it allows at least the User and Access layers to be standardized more while they transition the Resources (typically a pretty huge undertaking).
Workspace Service vs Platinum vs Workspace Suite
What is Workspace?
Workspace is better thought of as a concept- where everything comes together. This is what Citrix demonstrated at Synergy with the Workspace App. The concept was that instead of keeping all of these apps up to date in Receiver, Sync, etc you could have one app that aggregates both Citrix-Hosted and SaaS apps. People LOVED this concept, I think especially a lot of the administrators and engineers in attendance. And you know what- I get it. Less to maintain? Serviced in the Cloud so it can be managed from anywhere without all the outages? It makes sense. But this concept needs to be broken out a bit because it still confuses especially those of us who have been deploying on-prem resources for over two decades that aggregate cloud resources on occasion. Now we’re being asked to flip the concept and use the cloud to aggregate on-prem???
Refresher- What is Workspace Service?
Put simply, any Citrix Cloud service is merely taking what was an on-prem Control mechanism and moving it to the Cloud as a service instead of as a VM. So Citrix merely provides you with a console to manage the polices and settings, you don’t need to maintain VMs. As for connecting to your cloud locations for resources, this is done via a Cloud Connector- a universal software that sits on a Windows server in your cloud to create an SSL connection to the Citrix Cloud. This has the added benefit of eliminating VPN and Firewall needs. The connector acts as a bridge between specific services within the Citrix framework only. Citrix doesn’t need VPN access to your network; it only sees what you allow the Cloud Connector to see (DNS and Active Directory, along with the Citrix Resources).
As far as the Service itself? That’s what you pay for – whatever service you are consuming. In the case of Workspace Service- remember that it is the inclusion of all the primary Cloud services. If that is too much or not needed- there are other Services to which you can subscribe. Keep in mind that Citrix uses a monthly per-user figure for pricing, but you typically need to pay for at least a year in advance.
Platinum and Workspace Suite are On-Prem Control
I probably shouldn’t even be talking about Workspace Suite because it’s essentially dead IMO, but the concept was the same- a bundled license of products which in this case included some cloud-based components; but your Resources and Control for Apps and Desktops remained in your cloud exclusively. Platinum is still the best fit for most customers who push the edge of what can be used because it includes a boatload of useful stuff which I would get tired trying to go thru in detail. But for those that needed XenMobile as well as Platinum it made sense.
From the Citrix website: “The Citrix Workspace Suite is made up of the XenDesktop Platinum and XenMobile Enterprise products – inclusive of all the additional products included within, such as XenApp, XenMobile MDM, NetScaler SD-WAN (formerly CloudBridge), AppDNA and NetScaler Gateway user licenses.” In other words- don’t get it twisted: same name but very different functionality. The Workspace Service is much more all-inclusive and offers you the ability to run the control plane from the cloud. With Workspace Suite you administer both Control and Resource Layers.
What does all this mean to the average Enterprise customer?
Very little… for now. If I’m being honest – in my travels out in the world, most customers on Enterprise licensing aren’t even fully taking advantage of what can be done with what they have. I’m actually doing some private research on this and what I’m finding is that some are even aware of benefits their licensing already has such as AppLayering and Workspace Environment Manager… but they are not utilizing them at all! When I’ve dug into ‘why’ this is happening, in almost every case I’m finding that there was either simply no awareness of the entitlement or in more cases there simply isn’t the time or HR capital to manage said features. This is a shame because they are already paying for it. Now, I’ve begun addressing the problem as I can by launching a membership site where people can keep up to date and learn more about these kinds of things (and you should totally check it out!).
But I will admit this: Moving the Control and integration of these features to the Citrix Cloud makes a lot of sense. All Administrators will have to learn is how to use the consoles. Engineers will simply need to learn how to make it all fit together without having to do much of the grunt work, freeing them up their 1,713 other tasks. This has impacts on your Operations Layer! But you are essentially exchanging one cost for another… So is it valuable in that regard? Probably. But time will tell if we see any real movement there. Why?
THIS IS TOO EXPENSIVE! Well- I mean, sort of. In truth, it really isn’t when you look at what you are getting. But if you are not taking advantage of the features, then it totally is not worth it. That being said- if your organization doesn’t have the ability to fully support a mobile workforce both in the cloud and on-prem… this may be worth taking a look… when you’re ready. I’m not a fan of buying things because they are shiny. You really need to understand what you are getting! That said- be mindful of your spend on this- especially around upgrade season. If you are running 50 users and have no admin staff… does maintaining on-prem really make sense? Are you really getting the value out of it? From what I’m seeing in the field right now I’d have to say no; but there is still resistance to the price point. It’s a quandary for sure!
But consider this: I recently helped a customer with their upgrade to 7.15 LTSR. You know what took the longest? Dealing with their database issues! It added a whopping 10 hours of effort onto what should have been a relatively simple upgrade. So I decided to look at more of these upgrade projects. Another with issues with Windows services that disrupted the controller services. 20 hrs of effort. Another that had some massive issues with about every member of the Control layer… 80 hours of effort! Another with an issue of a sudden departing Citrix lead right before a major go-live- 80 hours + a few hundred more I couldn’t catalog. The vast majority would have been avoided, which is sad. The reality is that the skillsets aren’t there any more- hiring qualified Citrix help is extremely difficult right now. This means relying on consultants. So if you are paying someone $200+/hr this can really stack up in a hurry! I think people need to start putting this into their calculations of going to the cloud, because it really does significantly reduce how much you need to spend to maintain the control; you simply need to administrate and upkeep the Resources. Not saying that’s easy- but it’s less work by those needing specific qualifications that are hard to find. That is very significant.
I’m Here to Help
So- is your head still spinning? Mine too in some ways. But hopefully using some fundamentals helped in some way.
But I want your feedback! I’m considering doing a comprehensive, real-world course that would help better explain the Methodology and what is needed to maintain all of this both if you use Cloud and if you remain On-Prem or do the Hybrid approach discussed here. If you are interested- please contact me and let me know! I encourage you to join the membership site or even our free Facebook Group and chat with your peers and myself about it!
We’re back, talking more NetScaler Security! This is part of our four part series using my recommended Assess (or Understand), Design (Plan), Change (Build) and Maintain (Manage) Methodology.
In Part One we went into detail on how to assess and find just how secure your NetScaler configuration really is. We went thru a few of the very many leading practices and how to determine variances from them. If you didn’t read that one, go ahead and read it… I’ll wait. If you have gone thru the article and have your spreadsheet (or notes), we’re ready to continue! Also check back on it from time to time- I’ll be updating the article every now and again when new threats or methods emerge!
NetScaler Security Goals
First, let me say outright that we are not taking actions on this step. That will come in Part 3, Change. For now, we are making a plan of action- what do we intend to do in order to address the concerns we have from our Assessment. Setting some goals is key here.
Setting NetScaler Security Goals
Let’s go thru the list and determine what we want for the best balance between simplicity to deploy and good NetScaler security. We’ll start with the Highest Urgency and Importance items.
Score an A+ at SSLLabs.com. Looking from my example, I can see that my NetScaler Gateway is scoring a “C”. I think having an A+ will be a good indicator of acceptable NetScaler security.
Ban SSL3 and TLS 1. With TLS 1.2 supported and 1.3 on the horizon, my goal will be to sunset older SSL ciphers completely for better protection.
Secure the NetScaler Management Interfaces. I want to make sure that the leading NetScaler security practices are followed to prevent attacks. This will include defining ACLs and restricting access to the NSIP interface. And, of course- change the NetScaler’s NSROOT password.
Upgrade to the Best Firmware Choice. As I mentioned in Part One, just because a firmware update has been released doesn’t mean it is automatically best to use it. In my case, I want to be sure the firmware chosen meets minimums for NetScaler security, not features as my primary concerns.
Address Leading Security items. Our assessment indicated at least one leading practice that is security related, so we will validate those settings.
Fine Tune for XenApp and XenDesktop
Several lower priority findings and other items marked as Low Urgency and Importance.
Validate Leading Practices. As the Health Check Summary from CIS indicated there were several leading practices in need of adjustment from the defaults. I’ll focus on the ones that affect overall NetScaler security.
Address Functionality Items. Our assessment had a few items of low priority that may affect the functionality of other systems. In fact, in our case I’m going to even add one that I know needs to be addressed for functionality of the new EDT protocol.
In the Design phase, we will be determining Design Decisions – in this case declaring the configuration changes that will give us the best NetScaler security that makes sense for our use cases.
Score an A+.
I know that to get to A+ I have a lot of things to address, but I am going to first start with a pretty major caveat. Just as I noted in Part 1, we need to declare a minimum support for our SSL settings. If we go too strict, we lose the ability for certain endpoints to connect entirely. On the other hand, we may only want those with secure and updated operating systems to be able to connect at all. For our example, we know that the following is true:
We only support connections from up to date browsers and operating systems. All others are best effort and must meet NetScaler security minimums. That means no XP, Vista or out of date OS configurations. Sorry, for those of you stuck in 2008.
We do not have Thin clients in scope so the need to continue allowing TLS 1.1 to support older thin clients does not apply. We can also apply STS.
First- let’s deal with the same things I mention in my original NetScaler security article about getting the Best SSLLabs Rating (which will be updated from time to time)
First we will go with the most secure Cipher sets and configuration possible. Now- I know a few things going into this that I want to share with you:
First- upgrading firmware. In 2018, SSL Labs will begin downgrading scores for those that are vulnerable. When upgrading firmware, I typically start with the minimum I need and go up to the latest firmware as long as it is at least a month old. This is from experience in feature releases causing disruption for other functions. The most common I’ve seen is problems with AppFlow. If you’re using HDX Insight (and let’s be honest, you should be) this is something you need to be aware of. If you’re not using AppFlow… knock yourself out with the latest; odds are good you’ll be fine. But I can’t stress enough how important validation testing is when upgrading to new major releases (say, 11 to 12 for example). If you’re stuck or less confident- phone a friend. Or contact me using the form over to your right…
The way SSL Ciphers are named gets confusing in a hurry- but the names are organized in such a way that indicates a balance between security and performance. NetScaler Gateway has some pretty specific processing needs, especially when using a VPX which doesn’t have an offload chip. Therefore, the selections we’ll make as primary will be driven by performance first. This means in the case of DHE vs ECDHE, we’re going to favor ECHDE because they will perform better in our use case. If you don’t know what I’m talking about… that’s okay. I barely know myself if I’m being honest. But this what I have tested and deployed to over 50 NetScalers last year. Here’s more information if you are hungry for it: https://docs.citrix.com/en-us/netscaler/12/ssl/supported-ciphers-list-release-11.html Note again- that if you are using a FIPS Appliance, you’ll want to review this: https://docs.citrix.com/en-us/netscaler/12/ssl/fips-approved-ciphers.html The list gets complicated, but few care more about NetScaler Security than those using FIPS appliances. Best to understand what Ciphers will be supported for your appliance because the mileage varies!
We will make a set of Ciphers specific to NetScaler Gateway, which we will refer to as “NSG”
We will separate RSA from non-RSA keys for TLS 1.2 and above, because there is drama emerging there right now. To be flexible in the future, we’ll make it easy to disable the cipher set for testing.
Beware ROBOT. I did a re-scan of my SSLLabs assessment and found that at the end of Feb 2018, my NetScaler security score would be an F! This is due to the firmware being vulnerable to ROBOT attack. You can test for that vulnerability here: https://robotattack.org/. We will update the firmware as part of our process to deal with this threat.
Speaking of SSLLabs- another up and coming trend you need to be aware of is Perfect Forward Security (PFS). To explain this as briefly as possible, there are two ways to achieve PFS, by using DHE or ECDHE keys. Based on my limited testing and research, I have found that ECDHE performs better than DHE, though DHE does offer better security. If you are running an MPX or SPX appliance you may want to look into DHE (Diffe-Helman Exchange). I do not yet recommend this for VPX that are not hosted on SPX. A lot more information can be found here: https://support.citrix.com/article/CTX205282 But- as long as the ECC curves are in place (they usually are if you have any ECDHE ciphers bound, which we will), this is not a concern for you until the ‘rules’ change to favor DHE more heavily. You may also want to enable STS on your StoreFront servers if you have internal connections- but be aware that this may cause disruption to certain thin clients so TEST TEST and then TEST AGAIN!
Note- There are new Ciphers coming soon to support TLS 1.3 – for this article, however I am assuming that we will not have access to that firmware and I’ll update this later with those new sets. We’ll create a Cipher set name that we will keep updated as the latest and greatest and include “CurrentCiphers” in its name.
Other Recommended Settings
Remember that we are going to change our NSROOT Password. I thought about suggesting Active Directory integration for NetScaler as well but that is probably best for another blog post or referring you to other sites. If you’d like to learn more about the process and include it in your document, it is something I recommend as you won’t have your NSROOT password floating all over the place. https://support.citrix.com/article/CTX123782 Regardless- change your NSROOT password on a regular basis even if you have AD authentication configured.
Unless our testing reveals something different, we plan on implementing all the recommendations found in our Scout report. Rather than type them twice, I’ll list them in the design document below.
NetScaler Security Design Document
Pulling from all the items above, we can generate a document that lays out our plan for better NetScaler security so we can share it with others if needs be- but is also very useful if it is something that will take some time to get approved. It doesn’t have to be anything fancy, but I always advise people to write things down first – here’s something to get you started. PLEASE NOTE- these are just the items I found on my assessment example. You’ll want to include any findings you have as well!
Change to a complex password
Store the new password securely! I use Dashlane to store and securely control sharing of passwords with 2FA security. Sign up here to get $20 off for you and give $20 for me too! Way safer than putting them in a file!
Upgrade Firmware to latest stable within our major version release.
We’ll be using the 12.0 56.20 build in our example. I chose this simply as it is the latest at the time of writing and met the criteria of not being vulnerable to CVE-2017-14602 and 2017-17382.
Note- check with your Network Administrator to be sure the Window Scaling option will be supported (SACK is tied to Scaling.) If not, only list Nagle’s. Another key metric here is the Factor for Scaling. Typically 4 is correct but see the article at https://support.citrix.com/article/CTX113656 to be sure you will not need to adjust this value after conferring with your Network Administrator.
Drop invalid HTTP requests
Whenever possible, I believe in using a non-production test environment or components to validate changes before impacting users. Part of our Deployment Plan will involve testing these settings on a test NetScaler VPX prior to deployment in production. Once our changes are validated, we’ll set a time for a production outage so that we can roll back changes if required. For an HA deployment, it is very possible to do the updates in a way that minimizes production impacts. However, with NetScaler Gateway connections will be disrupted when we reboot. So whenever possible with this kind of update, I recommend declaring an outage or at the least setting the expectation that sessions may disrupted during a defined time window when the changes will be made.
So write out a deployment plan with a rollback plan. This is very useful especially when you must generate something for a Change Control board. Here’s the high-level plan:
Save the Running Configuration of the test NetScaler VPX
Upgrade NetScaler Firmware (if HA, use the advice in the article above to list out your steps)
Secure the NSIP Interface by introducing new ACLs (VERIFY before you continue)
Make the other changes noted above
Next time- I’ll guide you thru the process of actually making the changes! When we’re done, you’ll have NetScaler Security at a level of confidence that will exceed most enterprise customers I’ve been visiting lately!
The worst time to find out about problems in your Citrix environment is after they are already happening. But the built-in tools (Director) don’t really always paint the full picture to give you quality Citrix monitoring. You need Proactive Citrix Monitoring and ControlUp is here to help!
Citrix Monitoring Webinar on 11-29-2017
Yoni Avital (Founder & CTO of ControlUp) and I had an hour long CUGC Connect webinar hosted by the Citrix User Group Community! Wednesday, November 29th 2017 at noon Central (US).
Yoni first showed me ControlUp almost 4 years ago and I’ve been enjoying watching them improve every year. I have loved the single-pane-of-glass approach they take to environment monitoring and engagement.
Why View the Replay?
Aside from hearing my soothing voice moderate… Yoni demoed ControlUp 7.1 which I believe excels not only at Citrix monitoring (as in XenApp/XenDesktop) but NetScaler monitoring as well!
New features also include adding nVidia vGPU at the VM and process level to your Citrix monitoring, metrics of your published applications (yay!) and a new troubleshooting feature.
Will there be a Demo?
YES- you’ll be able to see this all in action live in the webinar.