Author Topic: Crowdstrike outage  (Read 6200 times)

Online Chuchum Ainer

  • Dansdeals Platinum Elite + Lifetime Gold Elite
  • ******
  • Join Date: Apr 2020
  • Posts: 764
  • Total likes: 411
  • DansDeals.com Hat Tips 1
    • View Profile

Online Chuchum Ainer

  • Dansdeals Platinum Elite + Lifetime Gold Elite
  • ******
  • Join Date: Apr 2020
  • Posts: 764
  • Total likes: 411
  • DansDeals.com Hat Tips 1
    • View Profile
Re: AA full ground stop?
« Reply #1 on: July 19, 2024, 02:46:02 AM »
Some friends on a flight scheduled to depart 10:33pm pdt. At right around then the pilot announced a paperwork delay, due to a computer issue, IT support en route. At some point they officially delayed to after 12. Then deplaned.

Offline ab613

  • Dansdeals Silver Elite
  • **
  • Join Date: May 2019
  • Posts: 56
  • Total likes: 8
  • DansDeals.com Hat Tips 1
    • View Profile
Re: AA full ground stop?
« Reply #2 on: July 19, 2024, 03:20:02 AM »
What I’ve been told, it’s a nationwide stop, effecting all airlines.

Offline AsherO

  • Global Moderator
  • Dansdeals Lifetime 25K Presidential Platinum Elite
  • **********
  • Join Date: May 2008
  • Posts: 32095
  • Total likes: 8943
  • DansDeals.com Hat Tips 79
    • View Profile
  • Location: NYC
Re: AA full ground stop?
« Reply #3 on: July 19, 2024, 03:23:00 AM »
Some Microsoft outage apparently, triggered by a flawed update to a Cybersecurity platform CrowdStrike that breaks their software and takes down systems using it.
DDF FFB (Forum From Birth)

Online Chuchum Ainer

  • Dansdeals Platinum Elite + Lifetime Gold Elite
  • ******
  • Join Date: Apr 2020
  • Posts: 764
  • Total likes: 411
  • DansDeals.com Hat Tips 1
    • View Profile
Re: AA full ground stop?
« Reply #4 on: July 19, 2024, 03:27:45 AM »
Some Microsoft outage apparently, triggered by a flawed update to a Cybersecurity platform CrowdStrike that breaks their software and takes down systems using it.
Just saw the same. Do I sell my $30 of CrowdStrike stock?

Much bigger than just airlines

Offline AsherO

  • Global Moderator
  • Dansdeals Lifetime 25K Presidential Platinum Elite
  • **********
  • Join Date: May 2008
  • Posts: 32095
  • Total likes: 8943
  • DansDeals.com Hat Tips 79
    • View Profile
  • Location: NYC
Re: AA full ground stop?
« Reply #5 on: July 19, 2024, 03:31:03 AM »
Just saw the same. Do I sell my $30 of CrowdStrike stock?

Much bigger than just airlines

There’s already a published resolution

https://x.com/anzdarrena/status/1814187916815433802
« Last Edit: July 19, 2024, 03:35:07 AM by AsherO »
DDF FFB (Forum From Birth)

Offline AsherO

  • Global Moderator
  • Dansdeals Lifetime 25K Presidential Platinum Elite
  • **********
  • Join Date: May 2008
  • Posts: 32095
  • Total likes: 8943
  • DansDeals.com Hat Tips 79
    • View Profile
  • Location: NYC
Re: AA full ground stop?
« Reply #6 on: July 19, 2024, 03:31:45 AM »
DDF FFB (Forum From Birth)

Offline Moshe Green

  • Dansdeals Lifetime Platinum Elite
  • *******
  • Join Date: Dec 2021
  • Posts: 1548
  • Total likes: 1308
  • DansDeals.com Hat Tips 1
  • Gender: Male
    • View Profile
  • Location: Israel
BSOD on startup - CrowdStrike Update mess-up
« Reply #7 on: July 19, 2024, 05:10:39 AM »
If someone has a computer that's getting a BSOD because of an Csagent.exe error, this was caused by a faulty update by security company CrowdStrike. This error has grounded so many flights/trains (the trains were already grounded, but you get the picture) and caused many headaches around the world.

The way to get around it is as follows.

1. Wait until your computer restarts a few times.

2. Troubleshoot it.

3. Choose different startup options.

4. Choose Safe Mode

5. Go to C:\Windows\System32\drivers\CrowdStrike

6. Locate and delete file matching "C-00000291*.sys"

7. Reboot normally.

Offline mevinyavin

  • Dansdeals Presidential Platinum Elite
  • ********
  • Join Date: Sep 2022
  • Posts: 2929
  • Total likes: 3494
  • DansDeals.com Hat Tips 5
  • Gender: Male
    • View Profile
    • Laptop help/deal request template
  • Location: Yerushalayim
  • Programs: Windows 10 and 11 Home, Word/Excel/Publisher 2021, Wordweb, McGill English Dictionary of Rhyme, VLC, Torat Emet, 7-Zip, Lightshot, Calibre, FDM, FreeFileSync, Teracopy, Handbrake, Lightshot
Crowdstrike outage
« Reply #8 on: July 19, 2024, 05:29:56 AM »
https://www.neowin.net/news/botched-security-update-breakes-windows-worldwide-causing-bsod-and-crashes/
Multiple companies worldwide are currently forced to suspend their operations due to a faulty cybersecurity update from CrowdStrike. The update is taking down thousands of Windows computers, causing them to boot loop and crash to a blue screen of death with the error message csagent.sys (PAGE_FAULT_IN_NONEPAGED_AREA).

Affected companies include banks, airlines, TV channels, and more, and some of them are forced to halt their jobs almost completely, with most Windows PCs not working due to the Falcon Sensor agent from CrowdStrike, a system that monitors network activity and prevents cyberattacks. One user from Malaysia said on Reddit that 70% of their laptops are not stuck in a boot loop:

Malaysia here, 70% of our laptops are down and stuck in boot, HQ from Japan ordered a company wide shutdown, someone's getting fireblasted for this shit lmao

CrowdStrike has already confirmed the problem and reverted the update. However, the machines that are already affected still cannot operate properly. While IT admins are scratching their heads in attempts to understand what happened and how to resuscitate their computers, a lengthy thread on Reddit suggests deleting a file in the CrowdStrike directory:

Workaround Steps:

Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
While booting into Safe Mode and deleting a single file does not sound too hard on a single machine, servicing hundreds of computers, remote devices, and cloud-based service will be quite a chore for IT admins.

Developing...

If someone has a computer that's getting a BSOD because of an Csagent.exe error, this was caused by a faulty update by security company CrowdStrike. This error has grounded so many flights/trains (the trains were already grounded, but you get the picture) and caused many headaches around the world.

The way to get around it is as follows.

1. Wait until your computer restarts a few times.

2. Troubleshoot it.

3. Choose different startup options.

4. Choose Safe Mode

5. Go to C:\Windows\System32\drivers\CrowdStrike

6. Locate and delete file matching "C-00000291*.sys"

7. Reboot normally.

Yet another website “down for maintenance” due to the CrowdStrike outage

https://www.reuters.com/business/aerospace-defense/american-airlines-issues-ground-stop-due-communication-issue-faa-status-page-2024-07-19/

What I’ve been told, it’s a nationwide stop, effecting all airlines.

Some Microsoft outage apparently, triggered by a flawed update to a Cybersecurity platform CrowdStrike that breaks their software and takes down systems using it.

There’s already a published resolution

https://x.com/anzdarrena/status/1814187916815433802

lol, probably not funny when you’re the passenger

https://x.com/s3pirion/status/1814186228524466614

https://x.com/jterryy07/status/1814197422169510222

Woot is also "down for maintenance."
Quote from: ExGingi
Echo chambers are boring and don't contribute much to deeper thinking and understanding!

Offline mevinyavin

  • Dansdeals Presidential Platinum Elite
  • ********
  • Join Date: Sep 2022
  • Posts: 2929
  • Total likes: 3494
  • DansDeals.com Hat Tips 5
  • Gender: Male
    • View Profile
    • Laptop help/deal request template
  • Location: Yerushalayim
  • Programs: Windows 10 and 11 Home, Word/Excel/Publisher 2021, Wordweb, McGill English Dictionary of Rhyme, VLC, Torat Emet, 7-Zip, Lightshot, Calibre, FDM, FreeFileSync, Teracopy, Handbrake, Lightshot
Re: Crowdstrike outage
« Reply #9 on: July 19, 2024, 05:40:20 AM »
https://www.techradar.com/news/live/windows-outage-july-2024-live


Though Amazon seems to be up right now. (Woot isn't.)
Quote from: ExGingi
Echo chambers are boring and don't contribute much to deeper thinking and understanding!

Offline mevinyavin

  • Dansdeals Presidential Platinum Elite
  • ********
  • Join Date: Sep 2022
  • Posts: 2929
  • Total likes: 3494
  • DansDeals.com Hat Tips 5
  • Gender: Male
    • View Profile
    • Laptop help/deal request template
  • Location: Yerushalayim
  • Programs: Windows 10 and 11 Home, Word/Excel/Publisher 2021, Wordweb, McGill English Dictionary of Rhyme, VLC, Torat Emet, 7-Zip, Lightshot, Calibre, FDM, FreeFileSync, Teracopy, Handbrake, Lightshot
Re: Crowdstrike outage
« Reply #10 on: July 19, 2024, 05:45:30 AM »
IL outage:
Hospitals Sheba, Rambam and Laniado
Meuchedet
Bank Hapoalim
MADA says phone service is somewhat affected
103 for electric company
Israel Post
Quote from: ExGingi
Echo chambers are boring and don't contribute much to deeper thinking and understanding!

Offline HudZ

  • Dansdeals Lifetime Platinum Elite
  • *******
  • Join Date: Nov 2019
  • Posts: 1988
  • Total likes: 1502
  • DansDeals.com Hat Tips 4
  • Gender: Male
    • View Profile
  • Location: Beis Medrash
  • Programs: Torah
Re: Crowdstrike outage
« Reply #11 on: July 19, 2024, 05:53:16 AM »
https://www.techradar.com/news/live/windows-outage-july-2024-live


Though Amazon seems to be up right now. (Woot isn't.)

And whatta ya know? DDF survives it all.

Offline mevinyavin

  • Dansdeals Presidential Platinum Elite
  • ********
  • Join Date: Sep 2022
  • Posts: 2929
  • Total likes: 3494
  • DansDeals.com Hat Tips 5
  • Gender: Male
    • View Profile
    • Laptop help/deal request template
  • Location: Yerushalayim
  • Programs: Windows 10 and 11 Home, Word/Excel/Publisher 2021, Wordweb, McGill English Dictionary of Rhyme, VLC, Torat Emet, 7-Zip, Lightshot, Calibre, FDM, FreeFileSync, Teracopy, Handbrake, Lightshot
Re: Crowdstrike outage
« Reply #12 on: July 19, 2024, 06:19:23 AM »
IL outage:
Hospitals Sheba, Rambam and Laniado
Meuchedet
Bank Hapoalim
MADA says phone service is somewhat affected
103 for electric company
Israel Post
Sharei Tzedek, Barzilai and Wolfson are also reporting affected. Health Ministry says that the hospitals trained for this eventuality and services are still being provided. However, ambulances were still told to prefer unnafected hospitals instead.
Ben Gurion is online but is being affected by delayed and canceled flights from elsewhere. El Al was unnafected.
Port of Ashdod is down.

In other news, UAE's foreign ministry is down.
Quote from: ExGingi
Echo chambers are boring and don't contribute much to deeper thinking and understanding!

Offline Essen est zich

  • Dansdeals Presidential Platinum Elite
  • ********
  • Join Date: Apr 2017
  • Posts: 2659
  • Total likes: 978
  • DansDeals.com Hat Tips 60
    • View Profile
  • Programs: Nichoach Vol 2
Re: Crowdstrike outage
« Reply #13 on: July 19, 2024, 06:41:23 AM »
Sysadmin on reddit is a fun place to browse right now.
Shloffen Shloft Zich

Offline yelped

  • Dansdeals Lifetime 10K Presidential Platinum Elite
  • *******
  • Join Date: Mar 2015
  • Posts: 12508
  • Total likes: 5050
  • DansDeals.com Hat Tips 45
    • View Profile
Re: AA full ground stop?
« Reply #14 on: July 19, 2024, 08:02:34 AM »
Pretty shocking that an update can make it out without smaller waves of testing first. Sounds like a few layers failed here.

Offline PlatinumGuy

  • Dansdeals Lifetime 10K Presidential Platinum Elite
  • *******
  • Join Date: Apr 2011
  • Posts: 15192
  • Total likes: 2493
  • DansDeals.com Hat Tips 11
    • View Profile
Re: Crowdstrike outage
« Reply #15 on: July 19, 2024, 08:43:59 AM »
I suspect this also knocked out Israel's air defense and enabled the drone to hit
״וזה כלל גדול: שישנא אדם כל דבר שקר. וכל מה שיוסיף שנאה לדרכי השקר – יוסיף אהבה לתורה.״ - אורחות צדיקים

Offline AsherO

  • Global Moderator
  • Dansdeals Lifetime 25K Presidential Platinum Elite
  • **********
  • Join Date: May 2008
  • Posts: 32095
  • Total likes: 8943
  • DansDeals.com Hat Tips 79
    • View Profile
  • Location: NYC
Re: Crowdstrike outage
« Reply #16 on: July 19, 2024, 09:01:18 AM »
I suspect this also knocked out Israel's air defense and enabled the drone to hit

Big (and shameful) if true. The only good news about that is that it wasn’t a cyberattack so they wouldn’t necessarily be able to time it next time.
DDF FFB (Forum From Birth)

Offline Yonah

  • Dansdeals Platinum Elite + Lifetime Silver Elite
  • *****
  • Join Date: Mar 2014
  • Posts: 652
  • Total likes: 414
  • DansDeals.com Hat Tips 4
    • View Profile
Re: Crowdstrike outage
« Reply #17 on: July 19, 2024, 09:32:18 AM »
I suspect this also knocked out Israel's air defense and enabled the drone to hit

The timing is not quite right. The crowdstrike outage started after the attack was over.

Offline Yonah

  • Dansdeals Platinum Elite + Lifetime Silver Elite
  • *****
  • Join Date: Mar 2014
  • Posts: 652
  • Total likes: 414
  • DansDeals.com Hat Tips 4
    • View Profile
Re: Crowdstrike outage
« Reply #18 on: July 19, 2024, 09:33:55 AM »
BTW - this is Microsoft's SECOND outage yesterday. The first one took out 1 of microsoft's 5 US regions of Azure cloud services. Frontier was grounded for hours before Crowstrike because of this, as were many other companies, and a lot of others (who use multiple regions) had downgraded services.

Offline Yehuda57

  • Dansdeals Lifetime Presidential Platinum Elite
  • *********
  • Join Date: Jan 2014
  • Posts: 5612
  • Total likes: 16226
  • DansDeals.com Hat Tips 6
    • View Profile
    • Squilled
  • Location: Brooklyn
  • Programs: Official Dansdeals salad correspondent
Re: Crowdstrike outage
« Reply #19 on: July 19, 2024, 09:41:19 AM »
I suspect this also knocked out Israel's air defense and enabled the drone to hit

Aside from the fact the strike was before the outage, the IAF already said they did track the drone but it was human error that issued the No Response.