Microsoft Teams has been a bit of a nightmare for a long time. This is because Microsoft wasn't following Microsoft's own rules and was installing the app per-user… in AppData! This is a profile management nightmare and always has been. The story is pretty much the same with OneDrive – a recent project of mine with Windows 10 1903 proved to be a challenge with added 30 second logons that couldn't be countered. But that all changed with the option of Per-Machine Installs.
You'll learn how to perform the per-machine install using native & AppLayering methods. Manuel includes scripts and some guidance for managing profiles as well.
Personally, and I'll be honest I never thought I'd say this… but I'm excited to see a practical means for replacing mapped drives and I think OneDrive in a Per-Machine install is exactly what will get us there.
My own opinions about this aside in terms of ethical hacking – a group claiming to be acting in the collective best interest of the world has released a code that exploits CVE-2019-19781 and starts mining bitcoin on the ADC.
UPDATE: I have put together a remediation checklist and a few more details! You can download the checklist now – head to https://ctxpro.com/?p=1493 for more details.
Oh, NOW I have your attention, huh… What started out as an interesting vulnerability in Citrix NetScaler / ADC code going back clear to version 10… just became a bigger problem. And many people were putting off fixing it until today.
Note the massive uptick in incidents. These people aren't doing anyone any favors or trying to make a statement. People are out to make a coin at your expense. But surely it won't end there so you really need to pay attention!
I (DJ) am working with some additional industry professionals to create a step-by-step course that you or your company will be able to purchase. The course will guide you through what we know so far, what you should know and how to remediate it. More importantly, the course will be updated and updates announced to purchasers and have ‘office hours' for support via chat. The course will include videos, examples and downloadable templates. The course will be offered thru the Citrix Hero Community, our free Citrix geek exclusive Mighty Network app. Pricing for the course itself has not yet been determined but will be announced by this weekend. To get notification of this offering, join the community or sign up for our email list and get a free e-Book.
What we know about CVE-2019-19781
The vulnerability affects all supported product versions and all supported platforms:
• Citrix ADC and Citrix Gateway version 13.0 all supported builds
• Citrix ADC and NetScaler Gateway version 12.1 all supported builds
• Citrix ADC and NetScaler Gateway version 12.0 all supported builds
• Citrix ADC and NetScaler Gateway version 11.1 all supported builds
• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds
Basically- it allows an attacker to place arbitrary code into portions of the ADC which can allow for a variety of badness to occur. Badness like grabbing password files or mining bitcoin (or whatever), possibly even attempting to create other backdoors.
One thing I will say, especially given how certain people have responded to this – is that this does not at all shake my faith in Citrix ADCs. Show me a company that has never had an exploit – you'll see that same company probably is too small to really matter. NetScaler / ADC deployment is massive, and the fact that it took this long to discover the issue at all speaks to the stability of the platform. Those calling for abandoning Citrix ADC are either acting out of fear, trying to profit from it, or are just jerks. No product out there is perfect or will never have an exploit found. Citrix is not ignoring this, though and neither should you. But don't be that person to use hurtful hashtags or spread fear rather than solutions.
Action Steps for CVE-2019-19781
Last update – 1.17.2020
Fix Script for Citrix NetScaler ADC
Use the instructions at https://support.citrix.com/article/CTX267679right now. The extended Citrix community is working on additional scripts. The easiest way to deploy these is to use Notepad ++ and PuTTY. I say this because you need to be aware of the way that your web browser will display quotation marks verses the way that the ADC will take it. Using Notepad ++ will help identify if you have a bad quote mark. Basically if one looks ‘upside down' from the other, you need to replace it with one from your keyboard. Fortunately – the fix is quick but does require a reboot to take full effect.
Some builds of NetScaler and Citrix ADC have not been properly applying the remediation patch due to a feature flaw that was patched in later builds. Full information from Citrix can be found here, but this looks to be specifically for builds In Citrix ADC and Citrix Gateway Release 12.1 build 50.28. You can logon to your ADC web admin page to verify the build – look in the upper right hand corner. I will be recommending an update regardless but if you are on this build you have to update for this to work, so I'd do so now.
If someone is cryptomining on your ADC- you'll see high utilization – but there's a catch. 100% is EXPECTED on newer versions of NetScaler/ADC.
Here's what do to. After you've run the prevention script and rebooted, get into the shell, or just enter
shell top -n 10
What you are likely to see is a process called NSPEE-00 or similar running at 100%. This is normal. What you DON'T want to see is other strange processes taking up a lot of CPU that stay that way. Knock on wood- so far I have not discovered any clients with active miners. But I have found a few that were compromised. To monitor continuously, just type in top without the -n 10. Once you're satisfied Ctrl-C will take you out of that.
However, in my mind, cryptomining is a secondary concern. Your company's information may have been exposed at some levels that have not yet been fully determined.
The big indicator of a compromise at this point is .xml files in directories they don't belong or have odd names. I will update this list soon but for now, look for some of the indicators noted at https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ Always run the workaround script first, but if you suspect you've been exploited, exporting your configuration and configuring from fresh firmware isn't a bad idea.
If you are compromised:
Take the ADC off the network.
Change the password of any LDAP or other AD/network accounts stored on the ADC.
Re-issue a new SSL Certificate and key file for any client SSL files on the appliance – the keys are stored in files that could theoretically have been read by the compromise.
If this is a VPX appliance, if you have snapshots of the machine prior to Jan 9th, 2020 it may be worth restoring that first but this is NOT A GUARANTEE of safety. My suggestion to be completely sure is to save your configuration file and restore it to a new VPX download.
Restore without starting – NOTE from the field: make sure your restore has the same Hardware address or your license will be invalid…
Disconnect the network before starting
Start the machine and verify using the console that the VPX does not appear compromised
Change the nsroot password
Attach the internal network only
Run the fix (alternatively- type this via the console to be safer)
attach the external network
Keep an eye on the logs
Replace SSL Certificates on the appliance at your earliest opportunity
Timeline and Updates
Jan 13 2020 Citrix has announced a timeline for ADC firmware that will include fixes.
Note- these are the initial timelines superseded on Jan 19th
Expected Release Date
31st January 2020
20th January 2020
20th January 2020
27th January 2020
27th January 2020
Jan 14 2020
I have started working with clients to remediate compromises and double-check other clients. I've updated some suggested quick things above.
I'm tracking reports from AWS users that if their nsroot password was not changed during deployment, it would expose their instance ID – if anyone can confirm this please let me know so I can update this. It is probably safe to assume at this point that any information stored in the ADC can be read by someone who knows what to look for. Change those passwords, people!
…more updates as I have validated them – there are a few additional remediation scripts being evaluated by other CTAs and CTPs especially.
eGInnovations – 7 Secrets to Becoming a Citrix Hero
On March 12th, I gave a presentation called “7 Secrets to Becoming a Citrix Superhero” on behalf of eGInnovations. You can register for the replay here. However, I have something special for you to grab before you watch the replay. I've put together a special printable guide you can use to walk thru the presentation! To get this emailed to you along with my Top 3 eBook (if you don't have it already), click here:
This is going to be an awesome day of knowledge share. And the best part? You don’t even have to leave your home or office to attend! I’ll be giving a report on the 12 “Worst Offenders” or my Dirty Dozen; the things I find most wrong during assessments. Due to the timeframe I’ll be cutting this down to a simple report and offering the full webinar separately. That said check out this lineup of other speakers and sign up! https://xenappblog.com/agenda/
Citrix Hero Program – Live Q&A
For Current Citrix Hero Program members- don't forget we're talking about PVS and MCS performance Friday, March 22nd from noon to 2 Central US!
The Nashville CUGC met on March 15th. It was a great time with my friends from Nutanix joining us.
What's up in April?
April is going to be a big deal! We will re-open the Citrix Hero Program to the public (get on the waitlist here) April 15th. I'm putting together quite the celebration around it which will have even MORE presentations and fun activities to get excited about! Join our email list for those announcements!
We will also be doing a special for Active Directory you won't want to miss!
“DJ, why are you giving away your best Citrix Tips?”
This question was asked of me recently- and I didn't hesitate to give an answer. So I think I'll give it to everyone here as a kind of open letter to the Citrix community…
It all has to do with quantity. I seem to give out the same things almost every week, not only in conversation but in my consulting engagements as well (regardless of if I'm representing Citrix, a Reseller or my own consulting company). So I asked myself two questions:
Why do I always seem to be giving the same advice? Is that such a bad thing? It makes my job easier, right?
Are they really that important if people aren't doing them?
Before I get into this behind-the-scenes look (or if you don't really want to know why, you just want my Top 3 tips right now) – click here to register for my free eBook where I give you those and put you on a monthly newsletter with even more tips – free:
The goal of this series is to outline some of the more common Citrix Mistakes that I have been seeing in my consulting engagements. These top three were chosen not because of frequency per se, but for the sheer impact they have. I chose them so that when you implement the fixes – you can be the hero. The #CitrixHero.
Fare the well, XenApp 6.5. Talk about an amazing run! But alas, the product is no more… I'll break this all down in a moment and what it means for you if you still find yourself running any XenApp version except 7.6 LTSR, 7.15 LTSR or 7.14 to 7.18. June 30th was a big day for Citrix, yet it passed with barely any fanfare. I thought maybe I'd do something about that!