Be A Citrix Hero During the Coronavirus COVID-19 Outbreak

Be A Citrix Hero During the Coronavirus COVID-19 Outbreak

Photo of DJ Eshelman from 2019

Whether you consider the responses thus far to “Coronavirus” to be panic or sensible precautions, one thing is sure: There hasn’t been a sudden demand for working from home worldwide like this … So how can you be the Citrix Hero during a season like this?

@TheCitrixCoach – D.J. Eshelman

I despise opportunity-based marketing. Yet, whether you consider the responses thus far to Coronavirus COVID-19 to be panic or sensible precautions, one thing is sure: There hasn’t been a demand for working from home worldwide like this since Bird Flu (or, was it Swine Flu?). Certainly not one this sudden and wide ranging. However, this case has been way worse in my observation; not medically but how our fear response has taken over. So how can you be the Citrix Hero during a season like this? If you have been listening to me on my podcast (Better Than You Found It) or reading posts on LinkedIn and such, you’ll be aware of a saying by which I live: You will not rise to the occasion. You will only rise to your level of preparedness. This holds true in EUC as much as in life.

You will not rise to the occasion. You will only rise to your level of preparedness. This holds true in EUC as much as in life. #CitrixHero Click To Tweet

Rising to the Coronavirus Occasion

If there is one thing I see consistently in Citrix Assessments – it is that the systems housing Citrix Virtual Apps and Desktop VMs (Single or Multi-Session OS) seem to be running past their maximums already. Adding on even more workloads suddenly is making this problem worse. Why? Because the IT systems were not PREPARED for this occasion. Time and again, I hear things like, “We’ll deal with that load when it comes.”

I hate saying I told you so.

How fast do you anticipate getting new servers given where the parts are manufactured? How will it get to you when delivery services start to become constrained? And if you believe the hype, how will you install them when you’re not allowed to go to work?

Citrix Hero Concept – Do More With Less

Something I talk about in my book Be A Citrix Hero is that with the right optimizations and design, you can do more with less. This is how YOU will be the Citrix Hero during the Coronavirus outbreak – you can set systems to perform at scale.

The “Be A CitrixHero” Book, Available on Amazon – https://ctxpro.com/book

Tweet This to your friends and do everything you can to let your boss know: The same optimizations that improve user performance can help you trade performance for scalability during a crisis.

The same optimizations that improve user performance can help you trade performance for scalability during a crisis. #CitrixHero Click To Tweet

By organizing resources properly (Chapter 3), optimizing the operating system to eliminate waste (Chapter 1), Utilize RAM properly for PVS and MCS workloads (Chapter 4), and using entitlements such as Citrix Workspace Environment Management (Chapter 6), the Citrix Hero is able to do more with less and be prepared for temporarily exceeding the design specifications.

I tell a story in Chapter 6 of an environment that I had done all of these things, and during an administrative glitch, they managed to shut down eight of their twelve 7.15 LTSR Server OS VMs. We had recently implemented just the basic settings I describe for WEM. Imagine their shock that all 300 users were on just four VMs, and no one had complained.

The Coronavirus scare is causing a similar scenario – suddenly, more users are on the system working remotely than before. Their expectations have not changed. They just expect to be able to work. Well, that and have ten browser tabs with Facebook, WHO, CDC, and various news outlets all open at once. So, are you prepared? There are some 40,000 words I wrote on the topic, so go get the book, so I don’t have to repeat myself. The good news is that what you implement during this crisis will set your company up to save money long term. Worth it.

Quick Coronavirus Citrix Hero Wins Not in the Book

I wanted to highlight a few things that are NOT in the book that you should be aware of as there is an increased need for users to work from home. Here are a few time and crisis tested ways you can rise to this occasion.

Implement RemotePC

Up to this moment, my go-to event for recommending RemotePC was during the Toronto floods. While systems were largely intact, many people could not get to their offices. Sound familiar? Oh, how quickly we forget this feature built into Citrix Virtual Apps and Desktops! RemotePC allows the Citrix Hero to install a light VDA onto the users’ existing PC and add them to a Delivery Group. Just like that, the user has remote access to their PC.

I fully recognize a primary challenge here: The Desktop Workstation has been dead or dying for a long time. Most desks today have a docking station for a laptop or a thin client (or PC converted into a thin client thanks to my friends at IGEL). I do NOT recommend installing RemotePC onto a laptop. They tend to be configured to go to sleep or save power. This is not typically a great combination. That said, in a pinch – having a user leave their laptop, pushing the RemotePC VDA along with some Group Policy to control the power state, and then removing all of that after the crisis may be viable.

The key with RemotePC is twofold in a crisis like Coronavirus (COVID-19):

  1. Provide Remote Access to company resources without additional risks, such as enabling VPNs. Because RemotePC utilizes your existing CVAD licensing, there are no added licensing or differences in external networking. This also means no added compliance requirements. It will, of course, consume a license – so keep that in mind.
  2. Lessen the burden on VDI and Server Desktops and provide continued access to user-specific applications and configurations.

Prepare for Increased Citrix Gateway Bandwidth Use

I’m going to ‘fanboy’ for just a moment. I have always loved that you can increase bandwidth allowed to a NetScaler (sorry… Citrix ADC) with just a different license key. No ripping out the appliance or having to deploy a completely new virtual machine. But what I do anticipate is that many companies out there may double, triple, or further increase the amount of staff accessing remotely to apps, desktops, or RemotePC. Each of these connections uses bandwidth – even when it is as highly optimized as Citrix protocols are.

You will want to keep an eye on your ADCs or set up monitoring to alert you if bandwidth is consistently exceeding 70% or more of your licensed limit. In some cases, the bottleneck may be external bandwidth. Either way, I have a quick win tip: Reduce the Bandwidth consumed per session. There are a few different ways to accomplish this, but first, there is an important concept to make sure management understands, and that is that the experience must be less rich.

  1. Reduce ICA Channels. Remember that each component configured that uses the ICA protocol will have a level of bandwidth assigned, and there is always a reserve, even if the components aren’t in active use. This means that if you can reduce the number of unused ICA channels, you can save bandwidth. Remember that these are user-context settings, by and large, so you can make reductions in just the users connected via Gateway with a policy. Here are a few that should be on the chopping block if you can work without them:
    • TWAIN redirection (what year is it?)
    • Port Redirection (it’s 2020)
    • USB Redirection (if not needed, which would be rare for remote access anyway)
    • Audio, especially microphone
    • Printing
    • Disk redirection & mapping
    • Clipboard redirection
  2. Restrict ICA Channel Bandwidth. This is an often missed setting where you can set a limit of bandwidth (or combined with our next tip a percentage of the overall session bandwidth) for certain channels that need to stay active. This has two benefits.
    • Reduces overall bandwidth for features that provide business value such as clipboard redirection or local drive mapping.
    • Prevents spikes in usage from exceeding your Gateway or connection maximums, which can cause pauses for other sessions. For example, if you set Printing to use a maximum of 500 kbit/sec, you will cause the print job to take longer but will not impact the other sessions as a result. During a crisis, we all make sacrifices. An extra 15 seconds to print doesn’t seem like much to ask.
  3. Restrict Session Bandwidth. Especially used with setting a percent of a session for channels, this setting can be great in constant-access scenarios. This setting is rarely used because to be effective, you must do a little bit of fortune-telling… you essentially need to predict how many active sessions will need bandwidth, padding against inactive, and all divided by the overall bandwidth available. This is not an exact science so don't treat this as an authoritative formula… but a quick and dirty way to figure out your initial settings. For example:
    • 500 active sessions
    • 100 inactive sessions (connected, just not actively using the screen) estimated at 1kbit/sec to maintain
    • 100Mbit Connection/Licensed Speed
    • 100,000,000-(100×1,000=100,000)=99,900,000 bits/sec
    • 99,900,000/500= 199kbit/sec per connection.
    • No problem. But make that 5000 active sessions, and we have a completely different dynamic.
    • Your session bandwidth restriction for connections over the Gateway should not exceed the math above us.
  4. The Big Win: Reduce Visual Transmission. This highly depends on the version of CVAD you are running, but there are a lot of options to reduce the bandwidth required to conduct this miracle we call remote display. All of these should be validated before simply pushing out via policy and very much keep in mind that they can be applied with additive filters (such as X user group when accessing thru a Gateway gets rule A, but Y user group in the same scenario may get rule B.)
    • Use Video Codec only for Actively Changing regions – this setting is time-proven.
    • If bandwidth is very constrained, set Do Not Use Video Codec. This is an “approach with caution” setting in my mind as it will often cause complaints. This setting will force Thinwire encoding, which can cause video to appear ‘choppy.’ Again – if we communicate with users that during the crisis they will have less capability, we can ready them to work differently.
    • Set a maximum framerate for the session. You can often reduce this to as low as 8 fps and still maintain a usable environment for task workers, but you may need 12 fps as a minimum. The point here is to set something, observe, and fine-tune.
    • Reduce color depth. Let’s face it, we all like the pretty, but going from 16-bit to 24-bit color was a huge jump. 32-bit color is a lot of information to have to send. I have, however, seen much less need to control this setting than in decades past. The codec is good at only sending what it has to. This is usually the last resort setting.
    • If you feel like going deeper into this, head over to the Citrix Tech Zone for a briefing.
  5. Avoid Handshakes During this Time of Crisis. Go with UDP based protocols if you can. Enlightened Transport is brilliant. After all – remember that we aren’t supposed to be shaking hands to prevent the spread of Coronavirus, so let’s take the same attitude as much as we can with ICA, shall we? A few caveats here:
    • Using UDP doesn’t reduce bandwidth; in fact, it may increase it slightly. This changes in high-latency situations where TCP causes retransmits.
    • Make sure your firewall has 443 UDP and TCP open to the Citrix Gateway VIP
    • Make sure your Gateway is configured for UDP using DTLS.

Focus on Quick Wins, Citrix Hero

There’s much more I can say here that gets way more complicated and take more time.

Now is not the time to get it perfect. That time was last month before people started working from home more, hoarding toilet paper but missing the canned goods. #CitrixHero #Citrix #Coronavirus #COVID19 Click To Tweet

I watch Citrix Engineers and Admins go down rabbit trails all the time, chasing blog after blog of perfect recommended settings. If you have the time to fine-tune – great. But the book Be A Citrix Hero, as well as this article, reflect my philosophy when dealing with times like dealing with Coronavirus, or any for which you may not be fully prepared. For example, this is not the time to try and implement complicated settings like Browser Content Redirection – even though it may be perfect for reducing server loads during times like this. If you have the time and can test it well enough, fantastic. Just keep a few things in mind for any changes you’re making during these times:

  1. Your Support Desk is probably already overwhelmed. Be their hero, not their villain.
  2. Focus on the highest positive impact in the shortest time
  3. Set expectations appropriately – Be Scotty, not _________ (insert politician full of empty promises here)
    • Don’t promise everything will be the same with a sudden increase in consumption. It probably won’t be; even if you take these steps, there will still be compromises to make.
    • Be extremely cautious, estimating timelines. Users that have not used remote access are now using it. This typically means a dramatic increase in handholding and trouble tickets.
  4. Don’t skip testing. I know the timeline is immediate. But if you have to stop everything else you’re doing to validate changes – do that. You will ruin more than just your day in a scenario like this by putting out untested best intentions. I don’t care what any blog or book, including my own, say – TEST BEFORE YOU DEPLOY, HERO.

I hope this was helpful to you.

Now go forth without fear. I know that in times like this, the fear of the unknown often takes over from the real problems in front of us. This is your chance to lead. Confidence is just as contagious as… well, come on, don’t make me say it.

Now for the marketing thing I’ll say and twitch while I’m typing it:

Working from home? Perfect time to pick up a copy of Be A Citrix Hero! No one’s watching – go for it!

I know, I know.

I’m the worst.

But I wouldn’t say all this if I didn’t genuinely believe that this will help you!

Be safe out there, and for the love of all that is holy, wash your hands!

PVS and MCS Performance: Memory Matters

PVS (and MCS) are amazing – but very often misunderstood – technologies. I have been in the reluctant camp with MCS (Machine Creation Services) mostly because of the huge impacts I have seen PVS (Provisioning Services) have on organizations workflow and performance. For many years the issue I find in the field is that people tend to focus completely on Storage performance with these technologies.

I won't spoil this for you. Storage is a minimal impact compared to the more modern ability to cache in RAM. RAM of the OS is always… ALWAYS faster than your storage can ever be. So while I do hold that Hyperconverged infrastructure does wonders for improving MCS (which has since the beginning suffered from poor read speeds and very poor deployment speeds) nothing improves performance like even a small amount of RAM caching.

But MCSIO (the caching in MCS) is unstable, you say… Only if you don't give it enough to work with and don't properly defragment your system. As I found in recent cases you'll get better results spending on RAM than storage… but let's not jump too far ahead.

This is a teaser article for Chapter 4 of the Citrix Hero Program.
The chapter given to the current subscribers is 19 pages, nearly 7000 words of content. What you see here is more of the “WHY” not all of the how. But I didn't want to leave you completely out in the cold. So I've included some links and other content that I hope will be helpful to you.

So what is the Citrix Hero Program?

Each month we tackle a new leading practice and dive in deep to make sure you understand it.
We have a live Q&A and video lessons to supplement the full chapter.
Members in good standing will have access to key past articles. This means that if you subscribe next month, you'll get access to this full article right away!

Enrollment will re-open in April – I encourage you to try it out then to lock in a special price!

Problem 1 – VDA Performance Lags

Problem Description

User experience suffers, especially later in the day. The problem seems to be worse for Machine Creation Services managed Server VDA hosted Desktops that are open all day long, however MCS Desktop VDAs seem to have similar symptoms. While not as high impact, a similarly configured Provisioning Services (PVS) target device exhibited similar symptoms. System pauses are experienced without subsequent CPU utilization spikes. While OS Optimization helps slightly the overall user experience seems to suffer regardless of how many programs are open. Workspace Environment Manager has helped memory and CPU issues but not solved the overall slowdowns issue.

Troubleshooting Notes

The Administrators note that the problems do seem to increase with additional sessions being active, but cannot explain when the same amount of sessions at an earlier point in the day are not experiencing the slowdowns so they suspect that the number of sessions may not be the primary factor.

Storage Engineers are concerned as IOPS do increase toward the end of the day, but are still way below what the system is rated to perform. A recent move to an all-flash array has not had a noticeable impact so they have encouraged looking more at the programs themselves or that it is a ‘Citrix problem’. Network Engineers report no issues or notable differences between beginning and end of the workday.

Solutions to Test

This problem tends to be caused because either the Cache In RAM with Failover to Disk feature has not been configured or has not been configured with adequate RAM. So… increase the RAM cache if you are experiencing these slowdowns and see if that helps.

Additional Considerations:

  • Defragment the master image prior to deployment
  • Assure Antivirus software does not have scheduled scans for non-persistent MCS or PVS-provided VDAs
  • Properly configure Citrix Optimizer (see Chapter 1 of the Become a Citrix Hero ebook – it's free!)
  • Use Citrix Workspace Environment Manager to control CPU and Memory resource priorities and to free up RAM, especially on Server OS VDAs

Want to go deeper? Every month we take problems just like this and dive deep into the solutions in the Citrix Hero Program. Enrollment opens April 15th.

Problem 2 – PVS Target Device (VDAs) lock up randomly or are slow at times

Problem Description

When PVS Target devices are configured properly (enough Cache, optimizations, etc) users are noting apparent lock-ups which clear after a few seconds. Programs are sometimes slow to load, especially early in the mornings. The problems are often much worse after new vDisk versions are deployed.

Troubleshooting Notes

Monitoring the VDAs indicate no unusual CPU activity- in fact at the times users note issues, Administrators may note that there is little to no CPU activity. Event logs do not indicate any OS configuration issues that are related.

The Citrix team confirmed that caching, optimization and the above recommendations have been configured properly. By further troubleshooting they found that many VDAs are indicating high retries.

Network Engineers note there are no packet loss issues or utilization issues on the network (note- be careful here; I have seen would-be Citrix Heroes not accept this explanation and blame the Network, only to later be embarrassed by the real cause…)

Solutions to Test

One of the most common causes of Retries is actually the PVS server not being able to meet read requests rapidly enough. Most typically this is caused by inadequate RAM on the PVS Server. When vDisks are read Windows Server will automatically cache the data. This reduces reads from disk because the non-persistent VMs are always reading the same data. However, because the cache is FIFO (First In First Out) if the amount of reads exceed the cache, the PVS server must read from it’s vDisk Store. Even with very fast storage, seek times and transfer can cause delays.

Citrix has guidance on how to get started with proper RAM sizing for PVS (see below). A common misconception is that it would be better to have faster storage- in fact some customers use physical hardware or dedicated Flash-Based or RAM-Based storage only to find that the improvement is minimal. RAM is far cheaper a solution. We’ll discuss in far more detail in the main lesson.

Want to go deeper? Every month we take problems just like this and dive deep into the solutions in the Citrix Hero Program. Enrollment opens April 15th.

Resolving PVS and MCS Performance Issues

So- How do you resolve these issues?

Here's a quick summary of what we cover in the rest of this lesson and some helpful links for you.

Problem 1 – VDA

Problem 2 – PVS Server

  • Recommended starting points for PVS (Server) RAM configuration: https://www.citrix.com/blogs/2013/07/03/pvs-internals-2-how-to-properly-size-your-memory/
  • Use Perfmon to identify vDisk reads are coming from disk instead of RAM – make sure your Cache Read Hits % stays above 80%
  • Antivirus scans can kill your performance… even/especially scans on the PVS server
  • Quick Tip: When you update a vDisk first boot a single VM, logon and launch all the programs normally launched; then start/restart other VMs
  • Versioning helps with the caching problem, believe it or not!

Want to go deeper? Every month we take problems just like this and dive deep into the solutions in the Citrix Hero Program. Enrollment opens April 15th.

Don’t Write a Book. Go Byte-Sized.

Don’t Write a Book. Go Byte-Sized.

Introducing the Byte-Sized Book Project

Writing a book is… really hard. But what if you could easily help one be written?

Well, that is exactly the opportunity in front of you!

Instead of writing chapters, editing, more writing, more editing and then distribution, worry, stress and…

okay, I may be relating some personal experience here…

As I mentioned in my February Newsletter – You can participate in a unique project… by submitting just 250 words.

This is the collaborative effort of Christiaan Brinkhoff and Bas van Kaam – the Byte-Sized Book Project.

The question is – do you have something you'd like to say about Cloud design principles, leading practices or even recommended reference builds you'd like to relay to the world? This is your shot to be heard!

This is all about creating value for the community as a whole, but not asking for a ton from each person. I think it's a brilliant approach!

So I encourage you to go for it! But don't wait- the plan is to get the book edited and done in the next few months!

c'mon- 250 words is easy!

Learn a lot more detail here.

Support the project by going to https://www.bookprojectbytesized.com/  – Follow them on Twitter and spread the word!!!


Christiaan Brinkhoff

Why I Gave Away My Top Citrix Tips

“DJ, why are you giving away your best Citrix Tips?”

This question was asked of me recently- and I didn't hesitate to give an answer. So I think I'll give it to everyone here as a kind of open letter to the Citrix community…

It all has to do with quantity. I seem to give out the same things almost every week, not only in conversation but in my consulting engagements as well (regardless of if I'm representing Citrix, a Reseller or my own consulting company). So I asked myself two questions:

  1. Why do I always seem to be giving the same advice? Is that such a bad thing? It makes my job easier, right?
  2. Are they really that important if people aren't doing them?

Before I get into this behind-the-scenes look (or if you don't really want to know why, you just want my Top 3 tips right now) – click here to register for my free eBook where I give you those and put you on a monthly newsletter with even more tips – free:

Register for the Free eBook & top Citrix tips



By With a Little Help from Our Friends

ByteSized Book logo