When monitors around the world display a “Blue Screen of Death” and you know it’s probably your fault, it’s got to be a terrible, horrible, no good, very bad day at work. That’s likely the situation inside CrowdStrike this weekend, as engineers at the cybersecurity provider struggle to recover from an update rollout that went very, very badly indeed. The rollout, which affected enterprise-level Windows 10 and 11 hosts running their flagship Falcon Sensor product, resulted in machines going into a boot loop or just dropping into restore mode, leaving hapless millions to stare at the dreaded BSOD screen on everything from POS terminals to transit ticketing systems.

Tales of woe from the fallout from what’s being called “the largest IT outage in history” are pouring in, including this very bewildered game developer who while stranded at an airport had plenty of ponder about why CrowdStrike broke the cardinal rule of software development by rolling a change to production on a Friday. The good news is that there’s a workaround, but the bad news is that someone has to access each borked machine and manually delete a file to fix it. Current estimates place the number of affected machines at 8.5 million, so that’s a lot of legwork. There’s plenty of time after the fix is rolled out for a full accounting of the impact, including the search for the guilty and persecution of the innocent, but for now, let’s spare a moment’s pity for the devs who must be sweating things out this weekend.

Back in 2011, Craig Fugate of the Federal Emergency Management Agency said of disaster response in the southern US, “If you get there and the Waffle House is closed? That’s really bad.” Thus was born the “Waffle House Index,” an informal measure of a natural disaster’s impact based on where individual restaurants in the chain that prides itself on always being open are actually up and running. With over 1,900 locations in 25 states, you’d think it would cover just about any emergency, but desperate Texans eschewed the index during the recent extensive power outages in the Houston area caused by Hurricane Beryl by inventing the “Whattaburger Index.” We haven’t had the pleasure of this particular delicacy, but it seems Texans can’t get enough of the hamburger chain, enough so that their online app’s location map provides a pretty granular view of a wide swathe of Texas. Plus, the chain thoughtfully color-codes each location’s marker by whether it’s currently open or closed, making it a quick and easy way to check where the power is on or off — at least during regular business hours. Hat’s off to the enterprising Texans who figured this out, and here’s hoping that life has returned to normal for everyone by now.

While we’re generally not fans of Apple products, which seem overpriced and far too tightly controlled for our liking, we’ve been pretty impressed by some of the results people have reported using their Apple AirTags to recover lost or stolen items — this recent discovery of a cache of stolen tools (fourth item) comes to mind. Results such as that require a “me too” response from the Android side of the market, resulting in the Find My Device network that, perhaps unsurprisingly, doesn’t appear to work very well. The test was pretty much what you’d expect — drop an Android-compatible tag in the mail along with an AirTag and track their journey. The Android tag only reported in a couple of times, while the AirTag provided a comprehensive track of the parcel’s journey through the USPS. Our first thought is that this speaks mostly to the power of being first to market, allowing Apple to have a more completely built-out infrastructure. But this may say more about the previously mentioned flexibility of Android compared to Apple; we know we noped the hell out of participating in Find My Device as soon as it rolled out on our Android phone. Seems like a lot of Android users feel the same way.

And finally, while we haven’t checked out comments on this week’s podcast, we’re pretty sure we’re getting raked over the coals for betraying our ignorance of and lack of appreciation for the finer points of soccer, or football. Whatever you call it, we just don’t get it, but we do understand and agree with our own Lewin Day’s argument that instrument-enhanced officiating isn’t making the game any better. Our argument is that in any sport, the officials are like a third team, one that’s adversarial to both of the competing teams, hopefully equally so, and that giving them super-human abilities isn’t fair to the un-enhanced players on the field/pitch/court/ice. So it was with considerable dismay that we learned that Major League Baseball is experimenting with automatic umpires to call balls and strikes behind the plate. While you may not care about baseball, you have to appreciate the ability of an umpire to stand directly in the line of fire of someone who can hurl a ball fast enough to hit a strike zone about the size of a pizza box the ball in less than 500 milliseconds. Being able to determine if the ball ended up in or out of that box is pretty amazing, not to mention all the other things an umpire has to do to make sure the game is played by the rules. They’re not perfect, of course, and neither are the players, and half the fun of watching sports for us is witnessing the very human contest of wills and skills of everyone involved. It seems like a bad idea to take the humans out of that particular loop.

In the past week, AT&T has announced an absolutely massive data breach. This is sort of a multi-layered story, but it gives me an opportunity to use my favorite piece of snarky IT commentary: The cloud is a fancy way to talk about someone else’s servers. And when that provider has a security problem, chances are, so do you.

The provider in question is Snowflake, who first made the news in the Ticketmaster breach. As far as anyone can tell, Snowflake has not actually been directly breached, though it seems that researchers at Hudson Rock briefly reported otherwise. That post has not only been taken down, but also scrubbed from the wayback machine, apparently in response to a legal threat from Snowflake. Ironically, Snowflake has confirmed that one of their former employees was compromised, but Snowflake is certain that nothing sensitive was available from the compromised account.

At this point, it seems that the twin problems are that big organizations aren’t properly enforcing security policy like Two Factor Authentication, and Snowflake just doesn’t provide the tools to set effective security policy. The Mandiant report indicates that all the breaches were the result of credential stealers and other credential-based techniques like credential stuffing.

Cisco’s Easy Password Reset

Cisco has patched a vulnerability in the Smart Software Manager On-Prem utility, a tool that allows a business to manage their own Cisco licenses. The flaw was a pretty nasty one, where any user could change the password of any other user.

While there are no workarounds, an update with the fix has been released for free. As [Dan Goodin] at Ars speculates, full administrative access to this management console could provide unintended access to all the rest of the Cisco gear in a given organization. This seems like one to get patched right away.

Bye Bye Kaspersky

Kaspersky Labs has officially started started winding down their US operations, as a direct result of the US Commerce Department ban. As a parting gift, anyone who wants it gets a free six-month subscription.

Just a reminder, any Kaspersky installs will stop getting updates at that six-month mark, so don’t forget to go on a Kaspersky uninstall spree at that time. We’ve got the twin dangers, that the out-of-date antivirus could prevent another solution like Windows Defender from running, and that security products without updates are a tempting target for escalation of privilege attacks.

Uncoordinated Vulnerability Disclosure

Let’s chat a bit about coordinated vulnerability disclosure. That’s the process when a researcher finds a vulnerability, privately reports it to the vendor, and together they pick a date to make the details public, usually somewhere around 90 or 120 days from disclosure. The researcher gets credit for the find, sometimes a bug bounty payout, and the vendor fixes their bug.

Things were not always this way. Certain vendors were once well known for ignoring these reports for multiple months at a time, only to rush out a fix if the bug was exploited in the wild. This slapdash habit led directly to our current 90-day industry standard. And in turn, a strict 90-day policy is usually enough to provoke responsible behaviors from vendors.

Usually, but not always. ZDI discovered the Internet Explorer technique that we discussed last week being used in the wild. Apparently [Haifei Li] at Check Point Research independently discovered the vulnerability, and it’s unclear which group actually reported it first. What is clear is that Microsoft dropped the ball on the patch, surprising both research teams and failing to credit the ZDI researcher at all. And as the ZDI post states, this isn’t an isolated incident:

microsoft: Exploit Code Unporoven

me: i literally gave you a compiled PoC and also exploit code

m$: No exploit code is available, or an exploit is theoretical.

me: pic.twitter.com/tIXJAbkRu4

— chompie (@chompie1337) June 12, 2024

While these are Microsoft examples, there are multiple occasions from various vendors where “coordination” simply means “You tell us everything you know about this bug, and maybe something will happen.”

Bits and Bytes

Claroty’s Team82 has documented their rather impressive entry in the 2023 Pwn2Own IoT contest. The two part series starts with a WAN side attack, targeting a router’s dynamic DNS. We briefly discussed that last week. This week is the juicy details of an unauthenticated buffer overflow, leading to RCE on the device. This demonstrates the clever and terrifying trick of attacking a network from the Internet and establishing presence on an internal device.

There are times when you really need to see into an SSL stream, like security research or auditing. Often times that’s as easy as adding a custom SSL certificate to the machine’s root store, so the application sees your forced HTTPS proxy as legitimate. In the case of Go, applications verify certificates independently of the OS, making this inspection much more difficult. The solution? Just patch the program to turn on the InsecureSkipVerify feature. The folks at Cyberark have dialed in this procedure, and even have a handy Python script for ease of use. Neat!

Speaking of tools, we were just made aware of EMBA, the EMBedded Analyzer. That’s an Open Source tool to take a look into firmware images, automatically extract useful data.

Breaking BSOD

Just as we were wrapping this week’s column, a rash of Windows Blue Screens of Death, BSODs, starting hitting various businesses around the world. The initial report suggests that it’s a Crowdstrike update gone wrong, and Crowdstrike seems to be investigating. It’s reported that renaming the C:\windows\system32\drivers\crowdstrike folder from within safe mode will get machines booting again, but note that this is not official guidance at this point.

Something super weird happening right now: just been called by several totally different media outlets in the last few minutes, all with Windows machines suddenly BSoD’ing (Blue Screen of Death). Anyone else seen this? Seems to be entering recovery mode: pic.twitter.com/DxdLyA9BLA

— Troy Hunt (@troyhunt) July 19, 2024