So, Crowdstrike!

In reference to events this week relating the the International IT outages across the World. Not entirely sure if this belongs in security, however I’m not sure we have a category that truly reflects the nature of this incident. I feel a little sorry for the company in question, this sounds like just a software bug and “these things do happen”.

“The Net”

Just a little historical recap to set the scene, back in 1995 (nearly 30 years ago now) Sony Pictures released a film called “the Net” starring Sandra Bullock and Jeremy Northam. It painted a picture of the perils of allowing a third party direct access to your systems with a view to keeping them safe. The fictional software in question was called “gatekeeper” and “the moral of this story” was, don’t trust other people with complete and immediate access to all your systems - because next time Sandra might not be available to save the day.

Now I know this was a work of fiction, however like many works of fiction it’s used to convey what should be a relatively obvious message. 30 years on, it seems that message didn’t sink in, despite many (many) people (ahem!) spending 30 years trying to re-inforce that message.

So, based on a widespread aversion to both which might be referred to as “common sense” and history, much like Thanos, the ensuing Chaos seen this week would seem to have been “inevitable”.

Why?

So if you install some software on your computer, then give that software full access to your computer (which is needed for some functionality like virus protection) and then allow that software to update itself, despite your best intentions, at this point you have lost control of your machine.

If you don’t think this is blindingly obvious, please comment below.
If you think this problem isn’t self-inflicted, again, please comment below.

If you’re still running anti-virus software on your computer … well, ignoring this specific incident for the moment, were you aware that Joe Biden just banned (before this incident) one of the leading anti-virus software suppliers in the US citing it as a potential security threat to national security.

If you’ve read this and STILL have anti-virus software on your computer, this your choice (!)

Joking aside, lots of software is self-updating?

Yes it is. Some of my Wordpress instances for example will automatically install security patches and major updates automatically. How is this different? Well my Wordpress instances auto-snapshot themselves hourly, so if there’s a problem of any kind, with a few clicks I can revert to a previous snapshot and disable the auto-update facility.

This is very different to having 300 checkout terminals in an airport which refuse to boot and/or connect to the network after an anti-virus update and require an engineer to visit each machine with a floppy disk or USB key to do a manual revert, restore or patch on each machine while tens of thousands of people wait.

Yes? Again, please comment if you disagree …

But we NEED anti-virus software, don’t we?

Well, my most recent experience with a Windows PC and anti-virus software is from maybe 15 years ago. At that point, if you were to call me and say “I seem to have a problem with my Windows machine, can you take a look”, and I were to ask “what sort of anti-virus software are you running?” and you were to say “none”, then I might expect to find maybe a hundred or so bits of malware of one kind or another installed on your machine. (mentioning no names but this was a specific example!)

So if you’re running Windows, I guess there’s an argument for saying yes.

Now you might ask “why” this is the case? In order to save a little time, let me point you at one of many available articles that might help.

What is the solution?

Well, anyone reading this probably already knows the answer, at least in part. Maybe a better question is, why are we still in this position given the solution is not only known but has been known for a long time.

What do you mean we already knew about this?

Well, a long time ago in a workplace far, far away, I used to get invited to a national Cyber Security conference aimed at educating businesses about the risks of Cyber crime and how to stay safe on the Internet. I was a little surprised to find that Windows was pretty much the only topic of conversation and there was literally Zero mention of Linux as a secure alternative.

So (!) I put in an FOI request (Freedom Of Information) to the relevant government department essentially requesting details about how much per month the government was spending on sponsorship of the event (and by implication, how much was being spent promoting various Operating Systems)

Strangely this did not go down well and I received a call requesting I withdraw the FOI request. Now based on my understanding of FOI requests I was a little spooked by this call, however it was promised that if I were to withdraw the request, Linux would receive a fair showing at the next event (10 months hence). Not wanting to push my luck, this sounded like a reasonable outcome, so I dropped it.

Needless to say the following year I didn’t get an invite and none of the online materials published after the event contained the word “Linux”. Shocker eh? At this point I noticed there were also a bunch of corporate sponsors for the event, mentioning no names but the top sponsor had a “$” in their name.

So, if government sponsored education is pointing in the other direction, what do you do?

What about Linux anti-virus software?

Yes it does exist, no, most people tend not to use it. Additionally, the anti-virus software you tend to see (“ClamAV” was the top Google result for me) isn’t really anti-virus software in the way a Windows user might understand it. Typically it’s just used to scan incoming emails for viruses, it’s not designed to be intrusive or to potentially go wrong and take your system out. (or render it inoperable) Indeed typically you would see it running on a server, not on an workstation.

I think my point is that *NIX in general (including Linux) was designed (from day 1) with security in mind. I guess this is partially a product of *NIX being designed as a multi-user mainframe Operating System, rather than as a stand-alone single user desktop Operating System.

General thoughts

My perception is that there are two things driving this particular issue.

  • Commercial marketing, sponsorship, influence, call it what you will
  • A general perception that commercial choices are risk-averse, “nobody ever loses their job by choosing M$”

I don’t see that you can do much with the first point. Companies with money are always going to try to use that money to proliferate their products, regardless of the comparative quality of their offerings.

On the second point, a time may come when large companies get hit sufficiently hard by outages like the ones we’ve seen this week, where choices perceived as historically “safe” might be called into question.

I just worry the country might be “finished” before this happens. Just to give you a “what if” … “what if” it wasn’t just 8.5M machines but 85M, and “what if” it wasn’t just a bug, what if those 85M machines were wiped, or wiped and then used to trojans against other machines. So “what if” we lost pretty much all of our IT for government, local government, the NHS, the police, the army, airlines, supermarkets (etc), for an extended period of time? (and indeed, what if this happened all over Europe?)

Just tin-foil-hat stuff tho’, right?

Very many thanks for this article. I shall promulgate the URL to all my friends.

As an aside, I have often asked organisations such as the Police, Neighbourhood Watch etc. why they spend so much time and effort telling us how to protect ones Microsoft system when Linux would be a good alternative, not requiring antivirus software. The silence was deafening!

I fear that you may be right in that if we don’t learn from this latest fiasco, then next time could be really catastrophic.

1 Like

With the amount of firms pushing for a cashless society this shows the importance of a back up system i.e. paper money in this case, so that things can still operate. Lots of businesses do not have a back up plan at all, as I read in an article by Brian Klaas of University College London

“The CrowdStrike debacle is a clear warning that the modern world is fragile by design. So far, we have decided to make ourselves vulnerable. That means we can decide differently too.”

Interestingly CrowdStrike will run on both Linux and mac but neither of these systems were affected by the update.

PS. In “The Net” Sandra Bullock played the part of Angela Bennett which would account for the name mix up!

Heh, yeah good catch, although in my defence I did get the name right on the second instance … :wink: … that’s me checking the date in Wikipedia and getting way too many names back :slight_smile:

Yeah, cash was never going away no matter how much the government would like to trace every penny we spend. But then if the till isn’t working I’m not sure the likes of Tesco would be able to tot up all the prices in order to take the cash.

Wasn’t aware Crowdstrike did a Linux version (but then I’d never heard of them before) but after 30+ years of not seeing the need, I’m not seeing anything different now. The thing that’s most worrying me at the moment in this context is supply chain attacks. Apart from developing everything in a container on a different machine (which is mostly what I do now) I’m not entirely sure how to combat this … it’s not something anti-virus software is likely to cope with.

Just noticed this;

I guess this week isn’t much better for them … :wink:

From the link:

We understand now that CrowdStrike’s software on Linux crashed due to a kernel bug involving BPF, which will need to be patched as per advisories from distro makers.

What is BPF and does the average Linux user need be concerned?

Mmm, I think BPF is network packet filtering, which kind of makes sense as it probably messes with your firewall.

If the a average user is running CloudStrike, then, well, I would be. My feeling is that the average user won’t know what CloudStrike is, in which instance, no, not at all :slight_smile:

There’s a long list of worthwhile things you can do to a Linux box to secure it and / or lock it down. IMHO installing “anti-virus” software typically isn’t one of them.

Thanks - that’s encouraging.

Linux has been so problem-free for me that I haven’t given much thought to intrusion security. Apart from using a firewall (which I have just started!) do you have any thoughts on basic technical precautions that are within the capacity of the average user? Things like containers and virtual machines are probably beyond me.

Naturally, I don’t push the envelope of daring web searches and don’t suffer from spam or phishing.

:slight_smile:

Well the easy start is that if you’re sat behind a residential broadband router, then you’re relatively safe on a Linux box by default.

Once you start installing “other” software on your machine, it’s worth “checking the doors” to make sure none are open or aren’t locked properly. To do this;

$ sudo netstat -natp | grep LISTEN
tcp  0 0 10.133.91.1:53  0.0.0.0:*   LISTEN   1926/dnsmasq        
tcp  0 0 127.0.0.1:631   0.0.0.0:*   LISTEN   92128/cupsd         
tcp  0 0 0.0.0.0:5355    0.0.0.0:*   LISTEN   820/systemd-resolve 
tcp  0 0 0.0.0.0:8602    0.0.0.0:*   LISTEN   875/tincd           
tcp  0 0 127.0.0.1:42385 0.0.0.0:*   LISTEN   97149/ssh           
tcp  0 0 0.0.0.0:655     0.0.0.0:*   LISTEN   878/tincd           

What you’re interested in are lines with something other than “127.0.0.1” in the fourth column, “0.0.0.0” means that the door is potentially open to the local network, so it’s worth knowing what it is. The last column (which is only populated if you run as sudo or root) should tell you which process had it’s hand on the door handle.

In this instance the only real item of concern is “ssh”, just make sure “/etc/ssh/sshd_config” has the “PasswordAuthentication” line set to “no”.

If you see a line with “0.0.0.0” and don’t know what it is, get curious, 99 times out of 100 it’ll be fine, however sometimes things get setup without the user realising … leaving a door ajar …

Another simple test is “lastlog” … just to see who’s logging in.

This is very helpful, although being able to interpret the o/p is obviously key.
Here is my output:

keith@E5570:~$ sudo netstat -natp | grep LISTEN
[sudo] password for keith:
tcp 0 0 0.0.0.0:4000 0.0.0.0:* LISTEN 7164/nxd
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 1345/hplip-printer-
tcp 0 0 127.0.0.1:5939 0.0.0.0:* LISTEN 6992/teamviewerd
tcp 0 0 127.0.0.1:25001 0.0.0.0:* LISTEN 7293/nxrunner.bin
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 990/cupsd
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 753/systemd-resolve
tcp 0 0 127.0.0.1:12001 0.0.0.0:* LISTEN 7211/nxnode.bin
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 7028/orbit_communic
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 971/sshd: /usr/sbin
tcp 0 0 127.0.0.1:23345 0.0.0.0:* LISTEN 6971/nxserver.bin
tcp 0 0 127.0.0.1:23344 0.0.0.0:* LISTEN 6971/nxserver.bin
tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN 7211/nxnode.bin
tcp6 0 0 :::4000 :::* LISTEN 7164/nxd
tcp6 0 0 :::8000 :::* LISTEN 1345/hplip-printer-
tcp6 0 0 ::1:7001 :::* LISTEN 7211/nxnode.bin
tcp6 0 0 :::1716 :::* LISTEN 3312/kdeconnectd
tcp6 0 0 ::1:631 :::* LISTEN 990/cupsd
tcp6 0 0 :::22 :::* LISTEN 971/sshd: /usr/sbin
keith@E5570:~$
(pasting has removed all the spaces - sorry)

There are several in the third column with all zeros, one of which is SSHD, but it’s my /usr/sbin so probably not of concern?
Strangely, Teamviewer and Orbit-comms are listed although not running (I thought).
Do you see anything of potential concern?

Ok, so teamviewer is running, if you look at the number 6992 next to “teamviewerd”, this is the process id of the teamviewer process.

ps ax | grep 6992

Should show you the process details, but it’s only listening on localhost not the network, so not really an issue other than cpu/memory usage.

Orbit has two daemons, the main process and the helper. If you visit http://localhost:8080 you will find the helper is running.

service orbit-helper stop  # stop the helper
systemctl disable orbit-helper # disable service @boot

Not a major issue, uses a tiny amount of resource and just a webserver.

hplip-printer … are you servicing print requests from your network? As it stands, anyone on your network (or anyone who can hack into your network via WiFi etc) has the potential to print to your printer via this print server. If you have no other machines, I’d reconfigure “cups” not to listen on 0.0.0.0 but rather limit it to localhost (127.0.0.1).

The only real concern would be “nxd”, which if I’m not mistaken is the “nomachine” server (?) This can run in client mode, server mode or both. Looks like you have it in either server or both, so it’s set up to allow people to remotely log into your machine. (obviously they would need to be on your local network and need your credentials, so not a threat so long as you’re aware it’s there and cater for it)

So behind a firewall, not too much of a problem depending on the NX login mechanism you have set up, passwords etc. However if you’re just using this for remote support, I’d reconfigure such that it runs in client mode only and doesn’t allow logins from other machines.

That’s very helpful - many thanks.

By chance, NoMachines has just updated itself, so I am aware of it. I think it was you who suggested it in place of Teamviewer since NX is direct machine-to-machine, although I haven’t used it yet. I’ll have a look at its setup.

Cheers.

[EDIT] just stopped the server as I couldn’t find anything like client mode.

Hi, yes I use NX a fair bit, but I always disable server mode :slight_smile:

So, main client screen;

  • settings → server → Ports, untick “Advertise this computer on the local network”

  • settings → server → Security, untick everything, but I think the first option is probably the only one you “need” to untick to disable the server.

That’s particularly helpful as one needs to be an NX expert to understand what all the options mean!

As my elderly Linux friends are gradually dropping off the radar I haven’t used Teamviewer for ages and haven’t had the opportunity to try NX, but it looks very useful so might have a go on the local network with another PC.

My thanks as always.

No problem.

I probably started using NX around 20 years ago. At the time the competition was really a choice between RDP (Windows thin client/Citrix protocol) or VNC. Probably the former was better (but M$ in origin), however both from my perspective have real limitations in terms of performance,

NX was a good solution that made a remote desktop realistic over moderate broadband speeds, so long as you didn’t do anything too unreasonable like streaming video in the client.

However (!) , since then “probably” a better solution has emerged, this is called “spice”, but there are two issues for me.

  • It’s not packaged with a nice UI like NX
  • I’ve not tried it from a security perspective in an Internet environment

If you look at virtual machines on Ubuntu / Debian and the “virtual machine manager” client, this uses “spice” to provide you with a graphic console onto your virtual machine. It’s performance in this context seems excellent (to me, it seems much better than NX) and it’s packaged up as part of the client software so there’s no configuration required.

https://www.spice-space.org/index.html

At some point I’m hoping someone sticks a UI on it with some security … at this point I would probably look at switching. (unless it’s out there already and I’ve not seen it?)

I had a look at Spice. Definitely not for me as:
a) it uses containers, which are beyond my comprehension, and …
b) from the description, implementing it looks really hard unless one is very competent at this sort of thing.

For experts, I think! but thanks for the info. I’ll try NoMachine one day, when I have time.
(I think we are a bit off-topic!)

  • It is a mechanism that can be used with containers to provide a graphical console to an instance of Linux desktop running inside a contain.
  • NX can also do this

But yes, I would agree it requires a degree of leg-work to make it run as a remote support option, which is why I’m still using NX :slight_smile:

You can download “Xspice” which is a spice server that should run on any Linux X machine, then connect to it with a spice client. Looks like most remote destops support spice these days.