fbpx
What Numbers Are You Tracking During COVID-19 Lockdown?

What Numbers Are You Tracking During COVID-19 Lockdown?

Numbers. Numbers. Numbers. Math. Math. Math.

Working from home, we look at numbers a lot. In the case of the COVID-19 lockdowns, we are probably looking more than we should. They get depressing in a hurry.
So here are some numbers that show a positve side of things – those coming from my friends (that's you!) in the Citrix Professionals community.

(this article was updated on May 8th, 2020)

The Citrix Heroes [email protected] Team

If you haven't heard yet – [email protected] is having a huge quarter. It's because they are doing research to determine effective treatments for COVID-19 that researchers are hoping will lead to a vaccine or at least drugs to help heal more rapidly (if you've ever heard of Herd Immunity you know how important this is in the long run to have more people getting it and recovering safely).
[email protected] installs on your computer and uses spare cycles of your CPU and GPU (the latter being a powerhouse) to contribute blocks of compute for the research. I have managed to give my Threadripper 2950 and AMD Radeon VII quite the workout this week, along with my old i7 3770k with a GTX TITAN. But I'm still completely outclassed in our little group.

Here's the breakdown (last updated 5/8/2020):

  • Our team # is 243,216
  • We're ranked 5620 of 250,736 (up from 12,929) – so we are now in the top 97% of all teams!
  • Homewerk is leading us all in credits, putting some nVidia GRID cards to use!
  • CTXPro_com (DJ's systems) climbed ahead with the addition of a RADEON VII card and a new tip: if you register for an account passcode, you get waaay more credits! But, the Vega II is an amazing card – when WU's are available, it's capable of 1.2 Million Points Per Day!

I'm really proud of you for stepping up and I'd love to see more of you join in! This research has never happened this fast – but it's a great use of background cycles you aren't using. Just don't install this into the datacenter or your work computers without asking permission, please.

%

Our Rank - Top Percentile

Who's Buying The Citrix Hero Book?

I have been watching these numbers fluctuate between countries. However, as of May 8th, 2020 here's how the top 3 break down:

Top Countries buying “Be A Citrix Hero” on Amazon

%

United States (down from 65%)

%

United Kingdom (up from 19%)

%

Germany

In addition to this is how many of you are taking advantage of the Kindle Owners Lending Library or Kindle Unlimited to read the book (that is an option if you want to read it quickly and don't want to actually own a copy!)
4448 Pages have been read so far in the lending library. (up from 2390 last month)

2070

in the U.S.A.
(up from 1166)

966

in the U.K.
(up from 592)

649

in Germany
(a new entry)

498

in India
(taking Canada's spot at #4)

Borrow the Book or Buy the Book

If you borrowed the book – show me what you've got! The more you read, the more Amazon helps get the word out about the book! If you got a copy with this method – I'd love to see 5000 pages by the end of the month! Get to reading!
Now you may be saying to yourself, “DJ, why would you be excited about people reading it for free?” – and honestly, I'm really excited that there are ways for you to get your hands on this material and boost your knowledge (and career). But if you sign up for Kindle Unlimited or Prime when you use the link at https://ctxpro.com/book – the site gets a bonus outside of book royalties (which average about 40%) when you do sign up. So it's a nice way to support the site without you having to spend any extra!

The Citrix Hero Fund

In the next week or so, look for a video message from me about this.
In January I had this idea – I talked about it with a few of my friends you probably know (Dane, Steve, Carl, etc). The concept is that we would as a community help get people unable to get into programs or to conferences that help you grow professionally- people that don't have corporate backing to do these things, for example.
Then – just before COVID-19 started dominating the news, a massive Tornado ripped thru an already somewhat impoverished part of Nashville – literally blocks from the capital.
My heart was broken – talk about a hard year!
So I decided to set up a small fund that I'm donating 30% of all of my royalties and earnings surrounding the Citrix Hero project.
We don't have exact numbers yet – but I can tell you that our first check should be arriving soon that we'll be donating to relief efforts here in Nashville. The intention is to do this until August.

After that – the fund will shift to a scholarship fund, hopefully by that point joined by some other projects I'm excited to tell you about.

If you're interested in donating to the fund, I will be getting you details soon but I'm really excited to get this going and I'll try to update you every month!

My goal – to raise $5000 for these efforts! Are you with me?

%

Of $5000 Raised

YouTube Subscribers

Three years ago, many of you suggested a YouTube channel. So we did – with a twist. We talk more about being a PERSON in IT, not an IT person.

But we need your help. YouTube doesn't take us seriously until we have 1000 subscribers. Help us get there!

https://ctxpro.com/YT

%

1000 Subscribers

Want even more numbers?

Stay tuned. I've got some results from a survey we have been conducting over the past 2 years that I think you'll enjoy reading. I've been so focused on writing that I haven't been able to finish the article but I hope to soon!
Which reminds me to remind you: We still want your opinion on where your career in End User Computing is heading! We are investigating the option of a Virtual Summit sometime this summer but we need confirmation that people are interested first!

You can take the survey herehttps://bit.ly/2JAKM1z

How Do I Become a CTA?

How Do I Become a CTA?

If you have a goal of being recognized as one of Citrix's CTAs (Citrix Technology Advocates) I have some specific advice for you to consider in this very first episode of Citrix Coach in a Car (getting coffee … or going to the chiropractor).

Continue the Conversation by ditching FB and joining our private social network for Citrix Heroes: https://community.citrixhero.com

Learn more about DJ and get his book on Citrix Leading Practices at https://ctxpro.com/book

Be sure and watch the video – or open in a new tab and follow along with the new flipbook feature!

Then, download the PDF and enjoy!

 

(note – DJ left the CTA program in 2020 to pursue additional interests but if you'd like to talk to him about how to achieve your goals find the contact form at the bottom of this article to schedule a time to do so)

Video Highlights

01:32: With the CTA program and the Citrix technology advocates, the goal really is to serve the community

01:47: What we're looking for is how much of a community participant you are. In other words, how much time you spend in the community as a whole, whether that be in Citrix user group meetings, whether you're leading those or helping out with those, are you active on the forums? Are you somebody who is contributing to CGC website, that sort of thing? What involvement do you have; whether it's your independent blog or you're contributing to other people's blogs.

01:47: What we're looking for is how much of a community participant you are. In other words, how much time you spend in the community as a whole, whether that be in Citrix user group meetings, whether you're leading those or helping out with those, are you active on the forums? Are you somebody who is contributing to CGC website, that sort of thing? What involvement do you have; whether it's your independent blog or you're contributing to other people's blogs.

03:01: To me add value means that you have made someone else's life improve, even if it's just a little bit.

03:14: A lot of people can be consuming content, other people can be creating content, and if you are in the add value space, usually you are trying to create content, that is, helping people with either a problem, or just sometimes it's brightening their day. 

03:32: [adding value] Other times it is, sharing something you've learned in the hope that it'll help someone else along the way.

03:55: Being a good member of the community is trying to build up rather than tear down.

04:14: We are looking for people that are very open and honest and are very much about the technology and how they can help people and makes the world a better place.

01:32With the CTA program and the Citrix technology advocates, the goal really is to serve the community

05:20: What we're really looking at for that program is how much you are actually involved in the community and how much value you're adding to other people's lives as a part of that and from Citrix's perspective.

05:40: Give good use cases for the technology that makes sense in the real world and that aren't just prompted by a salesperson or, or something along those lines.

Hope you enjoyed this video lesson! Be sure to subscribe to the YouTube channel for more!

Feel free to download the PDF by clicking on the link on the bottom of the flipbook above!

And please… SHARE this with your network!

Subscribe to the YouTube Channel!

Important: We need you to subscribe to our YouTube Channel! We have a few things we can only do if we get enough subscribers. Please take 15 seconds and subscribe!

Want some reading material that can help you better enable remote work? Well- we've got just the thing!

Grab a copy of “Be A Citrix Hero” today!

Contact DJ For Advice About Your Career

What To Do With Your Time During Coronavirus

What To Do With Your Time During Coronavirus

Who would have thought that all that practice helping people work from home would be so crucial to your company? So, now you're spending a lot more time at home. So… what to do with your time? In this special edition of Citrix Coach in a Car Getting Coffee… or going to the chiropractor, DJ gives some tips on how to manage your time!

Be sure and watch the video – or follow along with the new flipbook feature!

Then, download the PDF and enjoy!

Video Highlights

01:21: You will not rise to the occasion. You will only rise to your level of preparedness.
02:31: Even with the right things we can make do with less, and what that means is that we
can actually make some sacrices in user experience.
04:20: There's a couple of things that I wanted to recommend to you. One is, if you're not
already feeling prepared, get prepared. The other is to educate yourself.
10:28: You need to give yourself that time. Don't think that you will be able to just ignore it.
That's not how our minds work. That's not how we are built. We just are not built for
isolation as much as we think we are.
10:48: Even the most introverted people in the world still and ways to become connected to
others. We are social creatures. And so allocate time for that, along with work, along with
everything else. Allocate time to do that. But, here's the key: don't let yourself exceed that
time.
11:47: Write down goals for yourself.
12:09: You need to have some structure and you need to have some priorities set.
12:29: Add value right now.
13:15: Be the person who is showing up. Be the person who is making a difference in people's
lives. Connect with that. Really, really get into thinking about what it is you're doing
and how that's impacting others, because it is. Along with that, give yourself some time to
learn and grow. Give yourself some time to expand your knowledge
15:27: [goals] Write these things down during this time when you're at home because there
will be distractions
15:41: You make a plan, stick to it, but give yourself grace with these kinds of things. Don't be
unrealistic about what we're going to do.
16:01: Don't push yourself beyond what you can handle. But definitely celebrate your wins,
make sure you're spending some time learning new processesand new things. Absolutely give
yourself grace for things you want to get or you feel like you need to do as far as
social interaction when you're kind of cooped up at home. And then, like I said, make a plan
and stick to it.

Hope you enjoyed this video lesson! Be sure to subscribe to the YouTube channel for more!

Feel free to download the PDF —>

And please… SHARE this with your network!

PDF Transcript

Want some reading material that can help you better enable remote work? Well- we've got just the thing!

Grab a copy of “Be A Citrix Hero” today!

Be A Citrix Hero During the Coronavirus COVID-19 Outbreak

Be A Citrix Hero During the Coronavirus COVID-19 Outbreak

Photo of DJ Eshelman from 2019

Whether you consider the responses thus far to “Coronavirus” to be panic or sensible precautions, one thing is sure: There hasn’t been a sudden demand for working from home worldwide like this … So how can you be the Citrix Hero during a season like this?

@TheCitrixCoach – D.J. Eshelman

I despise opportunity-based marketing. Yet, whether you consider the responses thus far to Coronavirus COVID-19 to be panic or sensible precautions, one thing is sure: There hasn’t been a demand for working from home worldwide like this since Bird Flu (or, was it Swine Flu?). Certainly not one this sudden and wide ranging. However, this case has been way worse in my observation; not medically but how our fear response has taken over. So how can you be the Citrix Hero during a season like this? If you have been listening to me on my podcast (Better Than You Found It) or reading posts on LinkedIn and such, you’ll be aware of a saying by which I live: You will not rise to the occasion. You will only rise to your level of preparedness. This holds true in EUC as much as in life.

You will not rise to the occasion. You will only rise to your level of preparedness. This holds true in EUC as much as in life. #CitrixHero Click To Tweet

Rising to the Coronavirus Occasion

If there is one thing I see consistently in Citrix Assessments – it is that the systems housing Citrix Virtual Apps and Desktop VMs (Single or Multi-Session OS) seem to be running past their maximums already. Adding on even more workloads suddenly is making this problem worse. Why? Because the IT systems were not PREPARED for this occasion. Time and again, I hear things like, “We’ll deal with that load when it comes.”

I hate saying I told you so.

How fast do you anticipate getting new servers given where the parts are manufactured? How will it get to you when delivery services start to become constrained? And if you believe the hype, how will you install them when you’re not allowed to go to work?

Citrix Hero Concept – Do More With Less

Something I talk about in my book Be A Citrix Hero is that with the right optimizations and design, you can do more with less. This is how YOU will be the Citrix Hero during the Coronavirus outbreak – you can set systems to perform at scale.

The “Be A CitrixHero” Book, Available on Amazon – https://ctxpro.com/book

Tweet This to your friends and do everything you can to let your boss know: The same optimizations that improve user performance can help you trade performance for scalability during a crisis.

The same optimizations that improve user performance can help you trade performance for scalability during a crisis. #CitrixHero Click To Tweet

By organizing resources properly (Chapter 3), optimizing the operating system to eliminate waste (Chapter 1), Utilize RAM properly for PVS and MCS workloads (Chapter 4), and using entitlements such as Citrix Workspace Environment Management (Chapter 6), the Citrix Hero is able to do more with less and be prepared for temporarily exceeding the design specifications.

I tell a story in Chapter 6 of an environment that I had done all of these things, and during an administrative glitch, they managed to shut down eight of their twelve 7.15 LTSR Server OS VMs. We had recently implemented just the basic settings I describe for WEM. Imagine their shock that all 300 users were on just four VMs, and no one had complained.

The Coronavirus scare is causing a similar scenario – suddenly, more users are on the system working remotely than before. Their expectations have not changed. They just expect to be able to work. Well, that and have ten browser tabs with Facebook, WHO, CDC, and various news outlets all open at once. So, are you prepared? There are some 40,000 words I wrote on the topic, so go get the book, so I don’t have to repeat myself. The good news is that what you implement during this crisis will set your company up to save money long term. Worth it.

Quick Coronavirus Citrix Hero Wins Not in the Book

I wanted to highlight a few things that are NOT in the book that you should be aware of as there is an increased need for users to work from home. Here are a few time and crisis tested ways you can rise to this occasion.

Implement RemotePC

Up to this moment, my go-to event for recommending RemotePC was during the Toronto floods. While systems were largely intact, many people could not get to their offices. Sound familiar? Oh, how quickly we forget this feature built into Citrix Virtual Apps and Desktops! RemotePC allows the Citrix Hero to install a light VDA onto the users’ existing PC and add them to a Delivery Group. Just like that, the user has remote access to their PC.

I fully recognize a primary challenge here: The Desktop Workstation has been dead or dying for a long time. Most desks today have a docking station for a laptop or a thin client (or PC converted into a thin client thanks to my friends at IGEL). I do NOT recommend installing RemotePC onto a laptop. They tend to be configured to go to sleep or save power. This is not typically a great combination. That said, in a pinch – having a user leave their laptop, pushing the RemotePC VDA along with some Group Policy to control the power state, and then removing all of that after the crisis may be viable.

The key with RemotePC is twofold in a crisis like Coronavirus (COVID-19):

  1. Provide Remote Access to company resources without additional risks, such as enabling VPNs. Because RemotePC utilizes your existing CVAD licensing, there are no added licensing or differences in external networking. This also means no added compliance requirements. It will, of course, consume a license – so keep that in mind.
  2. Lessen the burden on VDI and Server Desktops and provide continued access to user-specific applications and configurations.

Prepare for Increased Citrix Gateway Bandwidth Use

I’m going to ‘fanboy’ for just a moment. I have always loved that you can increase bandwidth allowed to a NetScaler (sorry… Citrix ADC) with just a different license key. No ripping out the appliance or having to deploy a completely new virtual machine. But what I do anticipate is that many companies out there may double, triple, or further increase the amount of staff accessing remotely to apps, desktops, or RemotePC. Each of these connections uses bandwidth – even when it is as highly optimized as Citrix protocols are.

You will want to keep an eye on your ADCs or set up monitoring to alert you if bandwidth is consistently exceeding 70% or more of your licensed limit. In some cases, the bottleneck may be external bandwidth. Either way, I have a quick win tip: Reduce the Bandwidth consumed per session. There are a few different ways to accomplish this, but first, there is an important concept to make sure management understands, and that is that the experience must be less rich.

  1. Reduce ICA Channels. Remember that each component configured that uses the ICA protocol will have a level of bandwidth assigned, and there is always a reserve, even if the components aren’t in active use. This means that if you can reduce the number of unused ICA channels, you can save bandwidth. Remember that these are user-context settings, by and large, so you can make reductions in just the users connected via Gateway with a policy. Here are a few that should be on the chopping block if you can work without them:
    • TWAIN redirection (what year is it?)
    • Port Redirection (it’s 2020)
    • USB Redirection (if not needed, which would be rare for remote access anyway)
    • Audio, especially microphone
    • Printing
    • Disk redirection & mapping
    • Clipboard redirection
  2. Restrict ICA Channel Bandwidth. This is an often missed setting where you can set a limit of bandwidth (or combined with our next tip a percentage of the overall session bandwidth) for certain channels that need to stay active. This has two benefits.
    • Reduces overall bandwidth for features that provide business value such as clipboard redirection or local drive mapping.
    • Prevents spikes in usage from exceeding your Gateway or connection maximums, which can cause pauses for other sessions. For example, if you set Printing to use a maximum of 500 kbit/sec, you will cause the print job to take longer but will not impact the other sessions as a result. During a crisis, we all make sacrifices. An extra 15 seconds to print doesn’t seem like much to ask.
  3. Restrict Session Bandwidth. Especially used with setting a percent of a session for channels, this setting can be great in constant-access scenarios. This setting is rarely used because to be effective, you must do a little bit of fortune-telling… you essentially need to predict how many active sessions will need bandwidth, padding against inactive, and all divided by the overall bandwidth available. This is not an exact science so don't treat this as an authoritative formula… but a quick and dirty way to figure out your initial settings. For example:
    • 500 active sessions
    • 100 inactive sessions (connected, just not actively using the screen) estimated at 1kbit/sec to maintain
    • 100Mbit Connection/Licensed Speed
    • 100,000,000-(100×1,000=100,000)=99,900,000 bits/sec
    • 99,900,000/500= 199kbit/sec per connection.
    • No problem. But make that 5000 active sessions, and we have a completely different dynamic.
    • Your session bandwidth restriction for connections over the Gateway should not exceed the math above us.
  4. The Big Win: Reduce Visual Transmission. This highly depends on the version of CVAD you are running, but there are a lot of options to reduce the bandwidth required to conduct this miracle we call remote display. All of these should be validated before simply pushing out via policy and very much keep in mind that they can be applied with additive filters (such as X user group when accessing thru a Gateway gets rule A, but Y user group in the same scenario may get rule B.)
    • Use Video Codec only for Actively Changing regions – this setting is time-proven.
    • If bandwidth is very constrained, set Do Not Use Video Codec. This is an “approach with caution” setting in my mind as it will often cause complaints. This setting will force Thinwire encoding, which can cause video to appear ‘choppy.’ Again – if we communicate with users that during the crisis they will have less capability, we can ready them to work differently.
    • Set a maximum framerate for the session. You can often reduce this to as low as 8 fps and still maintain a usable environment for task workers, but you may need 12 fps as a minimum. The point here is to set something, observe, and fine-tune.
    • Reduce color depth. Let’s face it, we all like the pretty, but going from 16-bit to 24-bit color was a huge jump. 32-bit color is a lot of information to have to send. I have, however, seen much less need to control this setting than in decades past. The codec is good at only sending what it has to. This is usually the last resort setting.
    • If you feel like going deeper into this, head over to the Citrix Tech Zone for a briefing.
  5. Avoid Handshakes During this Time of Crisis. Go with UDP based protocols if you can. Enlightened Transport is brilliant. After all – remember that we aren’t supposed to be shaking hands to prevent the spread of Coronavirus, so let’s take the same attitude as much as we can with ICA, shall we? A few caveats here:
    • Using UDP doesn’t reduce bandwidth; in fact, it may increase it slightly. This changes in high-latency situations where TCP causes retransmits.
    • Make sure your firewall has 443 UDP and TCP open to the Citrix Gateway VIP
    • Make sure your Gateway is configured for UDP using DTLS.

Focus on Quick Wins, Citrix Hero

There’s much more I can say here that gets way more complicated and take more time.

Now is not the time to get it perfect. That time was last month before people started working from home more, hoarding toilet paper but missing the canned goods. #CitrixHero #Citrix #Coronavirus #COVID19 Click To Tweet

I watch Citrix Engineers and Admins go down rabbit trails all the time, chasing blog after blog of perfect recommended settings. If you have the time to fine-tune – great. But the book Be A Citrix Hero, as well as this article, reflect my philosophy when dealing with times like dealing with Coronavirus, or any for which you may not be fully prepared. For example, this is not the time to try and implement complicated settings like Browser Content Redirection – even though it may be perfect for reducing server loads during times like this. If you have the time and can test it well enough, fantastic. Just keep a few things in mind for any changes you’re making during these times:

  1. Your Support Desk is probably already overwhelmed. Be their hero, not their villain.
  2. Focus on the highest positive impact in the shortest time
  3. Set expectations appropriately – Be Scotty, not _________ (insert politician full of empty promises here)
    • Don’t promise everything will be the same with a sudden increase in consumption. It probably won’t be; even if you take these steps, there will still be compromises to make.
    • Be extremely cautious, estimating timelines. Users that have not used remote access are now using it. This typically means a dramatic increase in handholding and trouble tickets.
  4. Don’t skip testing. I know the timeline is immediate. But if you have to stop everything else you’re doing to validate changes – do that. You will ruin more than just your day in a scenario like this by putting out untested best intentions. I don’t care what any blog or book, including my own, say – TEST BEFORE YOU DEPLOY, HERO.

I hope this was helpful to you.

Now go forth without fear. I know that in times like this, the fear of the unknown often takes over from the real problems in front of us. This is your chance to lead. Confidence is just as contagious as… well, come on, don’t make me say it.

Now for the marketing thing I’ll say and twitch while I’m typing it:

Working from home? Perfect time to pick up a copy of Be A Citrix Hero! No one’s watching – go for it!

I know, I know.

I’m the worst.

But I wouldn’t say all this if I didn’t genuinely believe that this will help you!

Be safe out there, and for the love of all that is holy, wash your hands!

Install Teams and OneNote Per-Machine

Install Teams and OneNote Per-Machine

Microsoft Teams has been a bit of a nightmare for a long time. This is because Microsoft wasn't following Microsoft's own rules and was installing the app per-user… in AppData! This is a profile management nightmare and always has been. The story is pretty much the same with OneDrive – a recent project of mine with Windows 10 1903 proved to be a challenge with added 30 second logons that couldn't be countered. But that all changed with the option of Per-Machine Installs.

Fortunately, our friend and Citrix Technology Advocate Manuel Winkel has a step-by-step guide to publishing Teams and OneDrive Per-Machine! Check it out at https://deyda.net/index.php/en/2020/02/25/install-teams-onedrive-in-citrix-machine-based/

You'll learn how to perform the per-machine install using native & AppLayering methods. Manuel includes scripts and some guidance for managing profiles as well.

Personally, and I'll be honest I never thought I'd say this… but I'm excited to see a practical means for replacing mapped drives and I think OneDrive in a Per-Machine install is exactly what will get us there.

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

Remediation of Citrix ADC & NetScaler Vulnerability CVE-2019-19781

February 6, 2020

First things first – don't panic if you are one of the literally thousands of companies that were affected by CVE-2019-19781… because everyone was! But just because I say don't panic doesn't mean do nothing. I (DJ) have been working with several clients, experts and Citrix around solutions and I think we are in a place where most people have at least stopped the bleeding… but I grow concerned that many more have not.

Every. Single. Citrix ADC (NetScaler) was vulnerable and should be assumed to have been a target.

Some sentences are really hard to write. That last one was definitely one of them. But that all pales in comparison to the implications. The reality is- not only were several thousand companies not ready to handle this threat, my industry friends remain concerned that several thousand customers have not remediated because they didn't know how. Citrix and several sites including my own have given instructions – but if you don't know a few key things about the ADC and how to manage it outside of the GUI (Web Interface) then there is a confidence problem that has kept people from fixing the issue out of fear of locking out their users or making matters worse.

This has kept me busy at home and at hotel rooms for the last few months and I've come to realize I don't have the ability to help everyone. Here's what I can tell you:

  • Citrix ADC (NetScaler) VPX with a backup before December has been able to be patched and remediation performed in about 3 consulting hours.
  • Citrix ADC VPX with no backups have typically taken between 4 and 5 consulting hours.
  • Citrix ADC MPX (hardware platforms) are a particular challenge and in some cases have taken several evenings of downtime to resolve.
  • The skills needed to properly assess breaches, prevent them and fully remediate from the real threats of CVE-2019-19781 are those that are not commonly used, even by those that administrate Citrix environments.
  • Many customers deployed Citrix ADCs as a replacement to legacy Secure Gateways and have zero skills administrating NetScaler/ADCs, depending on the contractors that deployed them – contractors that often have moved on or are now so busy with these efforts on top of their existing workloads that it has become impossible to schedule help for many of them.
  • Worst of all – even though this has been a very popular and well documented case (my article “Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?” is the most popular article I've ever had. It eclipsed a nearly 3 year old popular article in a few weeks) many with Citrix NetScalers don't even know about the issue.

To help, I've developed a kind of a checklist/lesson plan, using my RiskLESS Methodology. Expand each section to see the detail of what needs to be done and the tools and knowledge you will need to be aware of to make it happen.

Understand

Understand the Threat

History of the issue

What is important to know about how the ADC works in this regard

What Hackers are doing (known threats)

What Citrix is Doing

Additional considerations

  • LDAP threats – high potential for network compromise and backdoors being setup up undetected
  • If LDAP account was also domain or elevated admin – critical to change password but also begin changing other domain accounts
  • SSL Key decode threats and need to re-key
  • NSROOT and any other local account passwords need to be changed

Detection

Citrix Tools
  • How to install the detection tool & use it
  • Web Admin Interface
  • Insight Services (CIS)
  • Shell bash scripts and python scripts
Command Line Inspection
  • Install PuTTY and SCP
  • Using SCP to download config + logs
  • Gathering breach info using the Citrix/FireEye Tool.

Documentation

Written Documentation
  • What you saw, when
  • Screenshots, etc
  • Document your findings of the exposure to management
  • Collect Passwords and store securely
    • NSROOT
    • LDAP account
    • myCitrix account
    • SSL provider
Remediation Considerations
  • Understand the remediation differences between VPX, MPX and SDX
  • If VPX/SDX – see if there are recoverable backups from before Dec 17th 2019 (if so, celebrate)
  • HA and Cluster considerations
  • Note to check all nodes for compromise – may make recovery easier
  • Firmware version considerations
  • Some versions of the firmware the remediation script did not work
  • Disclosure of breach and description to management
  • If LDAP account was elevated, severe danger requiring rapid disclosure to management and legal
Plan

Fast Remediation Steps

Determine what steps to take immediately and what will wait for a longer change window if this will be required (ie, MPX when access to the gateway will be needed during change window)

  • Emergency change considerations – brief outage for reboot
  • Apply remediation script and reboot
  • Create temp LDAP account password
  • Change LDAP account and/or password

Written Plan for Extended Remediation

  • Write out a plan of action based on your platform
    • Instructions for VPX
      • clean install
      • Plan for VPX recovery
      • Plan for HA secondary clean upgrade + fresh install primary
    • Instructions for SDX
    • Instructions for MPX
      • Special consideration for firmware ‘infection’
      • Optional recovery to trial VPX while recovering MPX
    • Instructions for changing passwords
    • Instructions for re-keying certs
    • Instructions and guidance for LDAP password and AD policies to deny interactive logon to that account (note- using “Domain Users” on network shares is a bad idea for this reason)

Change Management

  • Schedule outage(s)
    • Ideally- schedule complete Gateway outage by firewall rule blockage
    • VPX – estimated recovery time
    • MPX – estimated recovery time
    • Consideration – remediate now, update/upgrade firmware later
  • Communication plan (with users, management, IT)
  • Schedule staff or Consultants involved

Generate and store new passwords

  • USE A PASSWORD MANAGER – I use Dashlane
  • Any compromised password needs to be changed
  • Highly suggested that any person with Citrix Gateway access change their passwords

Download appropriate firmware

Change

During an outage and depending on your specific environment…

  • Run the remediation script

 During a more extended outage

  • Backup appliance firmware or VM
  • Backup running/saved config
  • Applying the configuration to new firmware or updating a clean backup
  • The Great Password Reset of 2020
    • LDAP (we recommend changing it again)
    • Service Accounts
    • Any account that has accessed Citrix via the gateway should be changed

Note – there are instructions on the Citrix website for all of this, but I'm betting if you were confident about them – you wouldn't have read this far. The good news is I think I can help. Scroll down if that's you after you read the final step.

Maintain

Monitoring

  • Command line monitoring
    • Httpaccess logs
    • top
  • Citrix tool to test for compromise
  • Web Interface watching for policy hits

Taking future notices seriously

You have a day or less. This is proof. Not to get preachy here but if there's one thing this whole event proves

Getting notified of future issues

Regular Tasks to schedule

  • Change LDAP password every month after breach for 3 months
  • Change NSROOT password every 90 days
  • Internal Health Check – watch for .xml files, etc
  • Health Check by qualified consultants

Get it Done NOW

It is my opinion that you need to be confident to do everything on this list and do it right now, if you haven't already.

The way I see it you have three options:

  1. Power thru it – spend the hours needed researching and making sure everything gets done. The good news is that there is plenty of information out there to help you get there- download the check list here.
  2. Hire someone like me to help. By all means, I'm willing to help. But as you can imagine I'm quite busy. As of this writing my first availability is in early March. You'll need 4 hours minimum and I charge a minimum of $185USD per hour ($740).
  3. Join my workshop on Feb. 17th. I'm taking a huge risk and setting aside a week, along with some of my friends and trusted advisors. We will walk you thru the process step by step and be available at regular times all week to answer your questions. The cost to join this workshop will be $399. If you are interested (or want your boss to pay for it) – see the contact form below and we'll get you in. Can't make it on the 17th? Don't worry – we'll be recording the sessions and I'll be available for office hours for your questions until March 30th, 2020 – and email after that (though hopefully you'd have fixed it by then!)

Grab my PDF Checklist

I put together a quick PDF document so you can make sure you've got your bases covered – download it here (no opt-in required)

Contact us for Help

Related Articles

Consulting

Consulting

A career option I have embraced is Consulting! In this episode I talk about three primary options: 1) Working for an agency (like Citrix Consulting, for example) 2) Working for a Partner/Reseller or Services Provider 3) Working independently Be sure and watch the...

Career Choices

Career Choices

So you've decided maybe you like this whole #Citrix thing. What are your #Career options? Join DJ Eshelman as he walks you thru his 4 (of 6) basic Success Lanes for IT roles in Citrix. What roles do you see yourself being most successful in? We'd love to hear from you...

A Simple Services Methodology

A Simple Services Methodology

In this episode, DJ explains his Services Methodology which will be featured in his new book "Just Do This" (summer of 2020). You'll learn how DJ has taken the methodology from companies like Microsoft, Citrix and others, combined it with some psychology (and...

Are People Mining Bitcoin on your NetScaler (ADC) using CVE-2019-19781?

My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.

UPDATE: I have put together a remediation checklist and a few more details! You can download the checklist now – head to https://ctxpro.com/?p=1493 for more details.

Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.

Fun little hacking for posturing is one thing. But now that money is involved, we see just how unethical this hacking really is. Manuel Kalloff has a great summary of the true nature of these impacts here: https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/

Note the massive uptick in incidents. These people aren't doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won't end there so you really need to pay attention!

You can also track this event here: https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/

(please note – these are all external links and I can't validate any content in them because of that)

I will try to keep updates to this page as much as I am able – but for now please realize that until the dates below every NetScaler ADC is vulnerable without the code… and possibly even then.

Update 1.20.2020

Citrix has increased their efforts on this remediation with new firmware. Initial releases are out now, with key releases like 12.1 following on Jan 24th. Blog with details here: https://www.citrix.com/blogs/2020/01/19/vulnerability-update-first-permanent-fixes-available-timeline-accelerated/

Update 1.18.2020

I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours' for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.

What we know about CVE-2019-19781

The vulnerability affects all supported product versions and all supported platforms:

• Citrix ADC and Citrix Gateway version 13.0 all supported builds

• Citrix ADC and NetScaler Gateway version 12.1 all supported builds

• Citrix ADC and NetScaler Gateway version 12.0 all supported builds

• Citrix ADC and NetScaler Gateway version 11.1 all supported builds

• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds

https://support.citrix.com/article/CTX267027
Update Jan 17 2020 – ” This vulnerability also affects certain deployments of two older versions of our Citrix SD-WAN WANOP product versions 10.2.6 and version 11.0.3 ” from https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

What this Citrix ADC Vulnerability Impacts

Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.

Update 1.20.2020 – I'd like to add some additional POSSIBLE impacts, including the possibility that ALL accounts that have gone thru the Citrix Gateway should have their passwords changed right away in addition to other potential impacts. Please review Thomas's article for additional information.

One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you'll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks.
No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don't be that person to use hurtful hashtags or spread fear rather than solutions.

Action Steps for CVE-2019-19781

Last update – 1.17.2020

Fix Script for Citrix NetScaler ADC

Use the instructions at https://support.citrix.com/article/CTX267679 right now. The extended Citrix community is working on additional scripts.
The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down' from the other, you need to replace it with one from your keyboard.
Fortunately – the fix is quick but does require a reboot to take full effect.

UPDATE 1.17.2020

Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I'd do so now.

https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

Check for Vulnerability in your ADC

The US Government has released a method of checking for this flaw. See https://www.us-cert.gov/ncas/current-activity/2020/01/13/cisa-releases-test-citrix-adc-and-gateway-vulnerability and https://github.com/cisagov/check-cve-2019-19781

1.17.2020 – Citrix also released this at https://support.citrix.com/article/CTX269180

Update – 1-14-2020

If someone is cryptomining on your ADC- you'll see high utilization – but there's a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.

Here's what do to. After you've run the prevention script and rebooted, get into the shell, or just enter

shell top -n 10

What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON'T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you're satisfied Ctrl-C will take you out of that.

However, in my mind, cryptomining is a secondary concern. Your company's information may have been exposed at some levels that have not yet been fully determined.

The big indicator of a compromise at this point is .xml files in directories they don't belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you've been exploited, exporting your configuration and configuring from fresh firmware isn't a bad idea.

If you are compromised:

  1. Take the ADC off the network.
  2. Change the password of any LDAP or other AD/network accounts stored on the ADC.
  3. Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
  4. If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
    1. Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
    2. Disconnect the network before starting
    3. Start the machine and verify using the console that the VPX does not appear compromised
    4. Change the nsroot password
    5. Attach the internal network only
    6. Run the fix (alternatively- type this via the console to be safer)
    7. attach the external network
    8. Keep an eye on the logs
  5. Replace SSL Certificates on the appliance at your earliest opportunity

Timeline and Updates

Jan 13 2020
Citrix has announced a timeline for ADC firmware that will include fixes.

Note- these are the initial timelines superseded on Jan 19th

VersionRefresh BuildExpected Release Date
10.510.5.70.x31st January 2020
11.111.1.63.x20th January 2020
12.012.0.63.x20th January 2020
12.112.1.55.x27th January 2020
13.013.0.47.x27th January 2020

Jan 14 2020

I have started working with clients to remediate compromises and double-check other clients. I've updated some suggested quick things above.

I'm tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!

…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.

Jan 16 2020

Citrix released an officially supported way to scan for the vulnerability, though it is not exactly user friendly https://support.citrix.com/article/CTX269180

Jan 17 2020

Citrix Posted this blog article clarifying a few things: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

DJ Eshelman announced that he will be working with a few other industry professionals to create a video and community support course offering to help people remediate this issue.

Jan 19 2020

Citrix blog post with updated firmware releases.

Citrix ADC and Citrix Gateway
VersionRefresh BuildRelease Date
11.111.1.63.15January 19, 2020
12.012.0.63.13January 19, 2020
12.112.1.55.xJanuary 24, 2020
10.510.5.70.xJanuary 24, 2020
13.013.0.47.xJanuary 24, 2020
Citrix SD-WAN WANOP
ReleaseCitrix ADC ReleaseRelease Date
10.2.611.1.51.615January 24, 2020
11.0.311.1.51.615January 24, 2020
I’m Writing a Book on IT Project & Services Methodology and I’d Love Your Help

I’m Writing a Book on IT Project & Services Methodology and I’d Love Your Help

I have learned a few things over the course of the last 20 years. These are things I've been teaching to other consultants, teams and coaching clients for years, so I've decided it was time to put them into a book!

The book focuses on what we call a Methodology: a repeatable process for working that is predictable enough for others to recognize and learn. Rather than a complex process that requires a certifiation to even understand, I have found that successful organizations use a methodology that can be understood by everyone from sales, services, support and up to the C-Suite.

In other words- if you're looking for another book on Agile or other DevOps methods… while some of those can be implemented inside of this – I have found that these methods tend to isolate IT and isn't always effective. Why? Because if your entire team isn't on-board, it tends to fall apart. So what happens when someone new comes in? Or a key person leaves? Or… the team gets lax in taking actions? What happens when moving too fast causes outages from risks that weren't properly identified? I call it a Resume Generating Opportunity. And it is exactly what I want to see people avoid.

That's why I have kept this book outside of the theoretical and describe EXACTLY what works NOW, and has been easy to understand everywhere I have taken it. The book is helpful if you are in sales, a service desk or a seasoned consultant. Everyone has something they can learn and I have plenty to teach!

 

I need your help! This survey will help me determine the best title and keywords to use to make sure everyone sees this book who needs it! I'd very much appreciate your vote!

EUC State of the Union 2019

There is a lot of marketing, hype and FUD (Fear, Uncertainty & Doubt) thrown around in the EUC (End User Computing) space. So I'm always grateful for true survey results to see the true trends in the industry. So, let's have a look!

You can register for and download the report here: https://vdilikeapro.com/announcing-the-state-of-euc-2019/

My friends Mark Plettenberg, Ruben Spruijt and Christiaan Brinkhoff have compiled this 73-page report that speaks the truth about the trends in VDI and EUC in general. Worldwide, just under 600 people turned in surveys which the team decoded and compared to previous years. Here are some of my personal favorite highlights.

Highlights of the 2019 State of the Union

  • Healthcare continues to be the largest business vertical staying on-premises for EUC.
  • Age of environments is interesting – VDI environments are often exceeding 5 years without significant updates. New designs had been declining but saw an uptick this year.
  • Citrix still dominates the VDI market but decreased by 7% to just over 50% of survey responses this year.
  • Nearly 22% are still using Windows 7
  • FSLogix use increased by about 2% – note that this was probably before people really knew they can get it for free now… But Citrix UPM is still the most popular choice.
  • PVS is still popular for imaging though usage is declining. MCS is almost neck-and-neck with PVS.
  • 39% of respondents were from North America with 50 overall countries represented.
  • The majority of companies have between 1,000 and 4,999 users
  • An overwhelming percentage (76%) are still using On-Premises Server Based Computing (RDSH or Multi-Session OS) – this only decreased by 2%.
  • VMware vSphere is still over 50% of the market, however, the big note I took from it was that Citrix, Microsoft and Nutanix Hypervisors are all increasing while VMware is decreasing. KVM (driven mostly by Nutanix) is up significantly this year.
  • More and more people continue to think of VDI as stateless or non-persistent.
  • 5.6% of respondents say they use Citrix EdgeSight for monitoring. If that doesn't shock you, it's overall position on the chart will. Not bad for being dead for nearly a decade.

I could go on- but honestly, I think you should download the guide and compare your results!