Having computer problems this fine July 19th, 2024 day?

Jstas

This is why:

https://www.cnn.com/business/live-news/global-outage-intl-hnk/index.html

Crowdstrike is an anti-virus and system/data security company run by a bunch of arrogant cowboys and managed by a bunch of chuckleheads.

Today's outage is affecting everything. Banks, airlines, airports, pharmacies, pharmaceutical companies, retail establishments, loan servicing companies and government offices around the globe. The UK's government was brought to it's computing knees.

It's being treated as a potentially intentional act.

The reality, though, is that Crowdstrike updated their software and since they use Microsoft software distribution channels to send their updates, they ended up pushing a bad change to their "sensors" around the globe and that change caused a problem that started cascading network events in organizations everywhere.

They say it's fixed and the fix is distributing but many companies are still down because the change they made affected critical infrastructure like domain controllers and authentication systems.

So if you can't get in to your bank today or your prescription is delayed being filled or even a shipment status is unavailable or "lost" blame Crowdstrike.

I'm going to sit here in my schadenfreude and resist as much as I can sending out "I told you so" emails to previous places of employment who decided to go with Crowdstrike because it was "better" (i.e.: cheaper). Then I think, though, that it's a waste of time because their email systems are probably still down.

muncybob

I was the 1st customer today at the garage we go to for the annual PA car inspection. He finished all the work and then said we have a problem. I was imaging a bill for some sort of repair but instead he said he couldn't connect to the state to enter my data, so I have to go back next week to get the sticker. Hopefully we won't otherwise be affected. Whenever somebody says "isn't technology great these days" I always reply...yea, when it works.

mrbigbluelight

I wish for our Benevolent Overlords to know that I, your most loyal and complient servant, will continue to observe and carry out all orders and commands to the best of my ability and will at all times and places observe for any signs of opposition to your Most Gracious Will.
One humble request, my Supreme All Knowing, All Seeing Overlords, please keep the Internet up long enough for me to finish downloading that Bailey and Kai video. 🙏
....asking for a friend 🥴

billbillw

My work laptop, which was brand new from Dell on July 3rd this year, is stuck in a BSOD loop because of this lovely Crowdstrike issue. Our IT folks are recommending a way to get into Safe Mode, but of course, they locked that out of the latest UEFI/Windows 11 Enterprise startup recovery menus.
I have no way of getting to Safe Mode. My laptop is bricked at the moment at a very young age.
Hopefully the smart guys that get paid more than I do can figure out a way to push this fix through on a machine that can't make it to a Windows welcome screen, and also without having to re-image my SSD. I literally just finished getting everything transferred from my old machine yesterday. I'll be really frustrated if I have to do it all over again.

Jstas

Depends, are you near an office?

If you're not, that's going to be difficult.

Makes things like CD-ROM drives and Floppy Disk drives seem not so stupid 'cause if you can get to the system BIOS page, you can change the boot device and get to the system that way so you can delete the driver and sys files that are causing the problem.

But, without the local admin privileges to do it, you'll need to have a domain admin or someone with the root Administrator account fix it.

Enjoy your 3 day weekend!

billbillw

3 day weekend? Pfft. I do believe they still expect us to work or pretend to at least .
I work from home, so I can access "most" of my required files through the cloud on my home PC which is unaffected (and most assuredly will never have anything from Crowdstrike installed). Most of my work files and resources are duplicated on OneDrive or Sharepoint and I can use the MS suite through Edge browser. I don't have my Zoom phone or Teams which is fine with me. I could install those on my personal devices, but I prefer to keep those programs limited to my work device.

sucks2beme

My last place of employment used crowdstrike. (Government).
I'm pretty sure things are chaos over there right now. Glad I'm
retired. I filled up my gas tank an hour ago. Pump transaction was
moving at a snail's pace.

Jstas

Wellllll.....about that.

Azure services are also impacted by an unrelated change to configurations that is affecting infrastructure Azure-wide and while recovery is in progress, the magnitude of the change is affecting progress expediency.

So your MS Suite stuff may have some hiccups in accessing Sharepoint repositories.

It's also affecting many hosted apps that use Azure for cloud hosting. That same update has also impeded the ability of companies to recover from the Crowdstrike fiasco.

Jstas

The UK Government...the news is being cheeky about it.

The UK Government is down, hard.

Parliamentary offices are completely offline, several social services offices are significantly impacted, I don't know about any military infrastructure but that tracks as I don't have a need to know and the NIS systems are having issues getting billing and approvals for medical care approved not just through the NIS system but also with 3rd parties like secondary insurance providers and pharmacy companies who can't seem to access what patients get what prescriptions.

UK is not the only ones having problems either. Australia, India, Pakistan and several middle eastern countries are having issues at governmental levels.

Additionally, United, American and Delta airlines all have global groundings issued so they aren't flying anything any where and there are hundreds of airports around the globe currently down and all services suspended.

The news I've been reading from Latin America and South America is even worse. Pretty much all of South America is at a standstill right now and most of Latin America is struggling too while the 4 major airlines based in Latin America are all on grounding orders too because they use the same systems as United and/or American and/or Delta.

This is really bad.

billbillw

Jstas wrote: »

Wellllll.....about that.

Azure services are also impacted by an unrelated change to configurations that is affecting infrastructure Azure-wide and while recovery is in progress, the magnitude of the change is affecting progress expediency.
So your MS Suite stuff may have some hiccups in accessing Sharepoint repositories.
It's also affecting many hosted apps that use Azure for cloud hosting. That same update has also impeded the ability of companies to recover from the Crowdstrike fiasco.

We moved away from Azure last year. All of our cloud stuff seems to be working fine.

Jstas

Well, it would since you're not using Azure anymore.

As long as you are managing your own OneDrive and Sharepoint installations and not relying on Microsoft to do it, you're unaffected if you didn't push any updates yet. Hopefully somebody shuts off your updates until this is all resolved.

Thing about this all is, though, the Crowdstrike problem is an easy fix, just delete C-00000291*.sys while in Safe Mode or WRE and then reboot normally. The offending update is already not available anymore so it won't download again. But, if you don't have the permissions to do that on your system you get to wait for someone who does to be able to help you.

msg

To what? (just curious. Infrastructure, data, collaboration tools?)

billbillw

msg wrote: »

To what? (just curious. Infrastructure, data, collaboration tools?)

Above my paygrade Scott. I recall some IT Teams meetings that I didn't attend something about an Azure to Isilon transition? I'm an environmental engineer who dabbles in home IT stuff but beyond that, I just hope things work like they should. Heck, maybe we do still use Azure. I just know my cloud stuff is working today.

billbillw

Jstas wrote: »

Thing about this all is, though, the Crowdstrike problem is an easy fix, just delete C-00000291*.sys while in Safe Mode or WRE and then reboot normally. The offending update is already not available anymore so it won't download again. But, if you don't have the permissions to do that on your system you get to wait for someone who does to be able to help you.

That was the instructions that came out from our IT group this morning. Unfortunately, they took local admin rights away from all of use a few years back. Prior to that, they let us smart computer folks have local admin because we had demonstrated that we were capable of not borking up our systems. No more. Also, the latested UEFI/Windows 11PE environment on my Dell Precision 3490 does not have any option to enter safe mode. I've tried all the tricks for hours this morning. Nothing gets it to boot past the PE recovery stuff.

Jstas

Isilon is a brand name for a type of on-premises SAN (Storage Area Network) system that is sold by the EMC corporation.

If you're running Isilon equipment, you're not using a cloud service or you are running your own cloud service.

EMC is wholly owned by Dell now, BTW.

I know there are cloud infrastructure services managed by EMC for Isilon customers but it's really just to allow Isilon customers to leverage global access points to keep users from having to backhaul data and connections all the way back to the on-prem SAN. They can, instead, use a more local access point and reach pre-cached or replicated data/apps that clients choose to put in the cloud services.

Jstas

billbillw wrote: »

Jstas wrote: »

Thing about this all is, though, the Crowdstrike problem is an easy fix, just delete C-00000291*.sys while in Safe Mode or WRE and then reboot normally. The offending update is already not available anymore so it won't download again. But, if you don't have the permissions to do that on your system you get to wait for someone who does to be able to help you.

That was the instructions that came out from our IT group this morning. Unfortunately, they took local admin rights away from all of use a few years back. Prior to that, they let us smart computer folks have local admin because we had demonstrated that we were capable of not borking up our systems. No more. Also, the latested UEFI/Windows 11PE environment on my Dell Precision 3490 does not have any option to enter safe mode. I've tried all the tricks for hours this morning. Nothing gets it to boot past the PE recovery stuff.

That's because of where your PE recovery utilities sit in the boot process. You can get to safe mode but not until the PE recovery utility is up. It's actually not a utility but a small, encapsulated OS that starts certain Windows processes to be able to mimic a Safe Mode or a WRE but you're not actually in the Safe Mode, you're in Windows but you have certain services suspended so that the WindowsPE utilities can make changes for you.

It's the same as SafeMode, you're just doing changes by proxy. But the problem with WindowsPE is that it doesn't have all of everything it needs encapsulated with it. It relies on certain Windows resources available from the locally installed OS. But if you can't start that local OS because of the issue causing the stop screen (actual name of the BSOD) then you can't start WindowsPE either.

In the past, to get around this while remote, I would pull out a Windows Repair disk or other recovery utility on a bootable USB or even a portable CD/DVD-ROM drive and boot off of that to fix the problem and then boot normally.

Lots of companies are restricting BIOS access now, too, though because it is a threat vector from a lost or stolen laptop.

msg

billbillw wrote: »

above my paygrade

Yeah, I realized that after I posted. Was just curious, generally speaking. I like to hear how other people are getting things done.

I'm seeing so much backward development these days, and I think it's dummifying people as well as being just plain counter-intuitive and impractical.

Some of it's also allowed more major players to tighten strangleholds. Ex., Broadcom, new evil owner exploiter of VMware, saying to small medical group begging for cost-increase relief, "VMware isn't for everyone." These people are the IT pros who helped these companies establish their footholds.

Guess we're all paying now for that "free stuff" from 2000-2014.

GD I'm becoming cynical. Lol
How did this happen?
I need more 2-wheel time!

[I struggled hard not to add "away from people" and "get me off this planet".]

msg

We were talking about this this morning. This just over from a colleague:

billbillw

Jstas wrote: »

That's because of where your PE recovery utilities sit in the boot process. You can get to safe mode but not until the PE recovery utility is up. It's actually not a utility but a small, encapsulated OS that starts certain Windows processes to be able to mimic a Safe Mode or a WRE but you're not actually in the Safe Mode, you're in Windows but you have certain services suspended so that the WindowsPE utilities can make changes for you.

It's the same as SafeMode, you're just doing changes by proxy. But the problem with WindowsPE is that it doesn't have all of everything it needs encapsulated with it. It relies on certain Windows resources available from the locally installed OS. But if you can't start that local OS because of the issue causing the stop screen (actual name of the BSOD) then you can't start WindowsPE either.

In the past, to get around this while remote, I would pull out a Windows Repair disk or other recovery utility on a bootable USB or even a portable CD/DVD-ROM drive and boot off of that to fix the problem and then boot normally.

Lots of companies are restricting BIOS access now, too, though because it is a threat vector from a lost or stolen laptop.

I'm not sure if my terminology is correct, but the options in the pre-boot do not seem to have any "startup settings" that would typically be used to start Windows in SafeMode.
I get this:


I'm seeing what I can do with a linux live USB or with Parted Magic on USB. (NM, neither worked)

UPDATE: Just got this from our IT:
What do you need to do?

Because local administrator rights are needed for most to facilitate the temporary fix, you will need to come into the office to fully execute the attached instructions. IT staff will be available to assist...


Eff that. I'm not driving into Atlanta on a Friday afternoon. If they don't figure out a way to do this remotely, my $hit laptop isn't going to be working until maybe Tuesday or Wednesday.

xschop

Cyber Polygon kick off today?

sucks2beme

Azure was also big at the city. They were a VMware bunch as well.
That must be getting pretty pricey to keep running by now.

Jstas

billbillw wrote: »

I'm not sure if my terminology is correct, but the options in the pre-boot do not seem to have any "startup settings" that would typically be used to start Windows in SafeMode.
I get this:


I'm seeing what I can do with a linux live USB or with Parted Magic on USB. (NM, neither worked)

UPDATE: Just got this from our IT:
What do you need to do?

Because local administrator rights are needed for most to facilitate the temporary fix, you will need to come into the office to fully execute the attached instructions. IT staff will be available to assist...


Eff that. I'm not driving into Atlanta on a Friday afternoon. If they don't figure out a way to do this remotely, my $hit laptop isn't going to be working until maybe Tuesday or Wednesday.

You're not going to get an actual SafeMode from WindowsPE.

WindowsPE can fix the same stuff that you can in SafeMode but it doesn't need to be in SafeMode to do it.

BUT

If you don't have the account permissions to do what you need to do, you won't see the options on that screen.

Given the response from your IT weenies, this is your case.

I hate WindowsPE anyway. Just makes things more difficult and sometimes ineffective.

One thing about using other OS bootdisks, if you have a drive encryption service running, if it doesn't come up before your bootable partition does, you're not going to be able to read the drive.

Even if it does come up, though, unless you can exactly duplicate your user account on the local system, it's not going to let you do anything. And, again, judging by what your IT weenies are saying, even if you could authenticate your user account, you won't have the permissions to do it anyway.

Maybe try the keep rebooting trick? Otherwise, enjoy your drive to the office.

billbillw

So apparently, my newly refreshed PC is presenting a much different set of options compared to everyone else in our Department. Sheesh. I'm going to try and go in Monday morning. Ugh.
Most people are offered a Startup Options that can do SafeMode with networking and CMD. I don't get that. Also most people were able to select a System Restore that allowed them to revert earlier this week. Doesn't work on mine

Jstas

Hmmm...that seems like a botched image deployment on initial setup or, they have an improperly configured security group that you're in.

I'm not going to try and fix it from here. Your IT team needs to figure out what the deal is but in all honesty, it's still a fresh system and you will have to move everything over again but I would have them back up the user files, dump them out to OneDrive for you or something and re-image it clean. It will give you nothing but frustration if they "fix" it to get it working as it will be an outlier.

If it's not in the proper groups where it's supposed to be, every software update will go like this for you. You won't get things that you're supposed to have because your Policy Group didn't have you system in it's distro list and that will mean that something like Forescout or Symantec Endpoint Protection or Trellix will kill your access to the network until it gets fixed. That will happen over and over again, probably on a quarterly basis, until your system gets a clean install and put in the proper security and policy groups.

skipshot12

Hell yea… let’s also take the world’s, and the US dollar, to a complete digital system. Get rid of cash all together.
Don’t see an issue there!

Jstas

Something interesting for y'all to ponder.

There was an event in 2010 called "The DAT 5958 Update" that then McAfee foisted on the general public. It was bad. Essentially a bad virus definition file from McAfee labeled the scvhost.exe file as a threat and caused the McAfee AV software to remove it on any systems running Windows XP Service Pack 3. It was bad and it pretty much caused similar problems then. But it only really affected folks who were on unmanaged networks at the time because that was the only place McAfee Agents could get the automatic update from McAfee. Lotsa regular people lost lots of data and it cost many a lot of money to recover.

Here's more detail on it:

DAT 5958 update
On April 21, 2010, beginning at approximately 14:00 UTC, millions of computers worldwide running Windows XP Service Pack 3 were affected by an erroneous virus definition file update by McAfee, resulting in the removal of a Windows system file (svchost.exe) on those machines, causing machines to lose network access and, in some cases, enter a reboot loop. McAfee rectified this by removing and replacing the faulty DAT file, version 5958, with an emergency DAT file, version 5959 and has posted a fix for the affected machines in their consumer knowledge base. The University of Michigan's medical school reported that 8,000 of its 25,000 computers crashed. Police in Lexington, Ky., resorted to hand-writing reports and turned off their patrol car terminals as a precaution. Some jails canceled visitation, and Rhode Island hospitals turned away non-trauma patients at emergency rooms and postponed some elective surgeries. Australian supermarket Coles reported that 10% (1,100) of its point-of-sales terminals were affected and was forced to shut down stores in both western and southern parts of the country. As a result of the outage, McAfee implemented additional QA protocols for any releases that directly impacted critical system files. The company also rolled out additional capabilities in Artemis that provide another level of protection against false positives by leveraging a whitelist of hands-off system files.

I told you that because, in 2010, the Chief Technical Officer of McAfee Corporation was a guy named George Kurtz.

The Founder and CEO of Crowdstrike is also named George Kurtz.

That's because they are the same guy.

https://en.wikipedia.org/wiki/George_Kurtz

billbillw

Gotta love the modern corporate CEO reward system. Ruin a company, take golden parachute payout as it goes down in flames. Wait a few years, rinse and repeat. I just wish these people would get fired and never be hire-able again like the rest of us would if we oversaw such a failure.

Jetmaker737

Wow. This @ssclown should have his "tech company executive" license revoked. What an @sshole.

billbillw

Jstas wrote: »

Hmmm...that seems like a botched image deployment on initial setup or, they have an improperly configured security group that you're in.

I'm not going to try and fix it from here. Your IT team needs to figure out what the deal is but in all honesty, it's still a fresh system and you will have to move everything over again but I would have them back up the user files, dump them out to OneDrive for you or something and re-image it clean. It will give you nothing but frustration if they "fix" it to get it working as it will be an outlier.

If it's not in the proper groups where it's supposed to be, every software update will go like this for you. You won't get things that you're supposed to have because your Policy Group didn't have you system in it's distro list and that will mean that something like Forescout or Symantec Endpoint Protection or Trellix will kill your access to the network until it gets fixed. That will happen over and over again, probably on a quarterly basis, until your system gets a clean install and put in the proper security and policy groups.

Not sure what the issue is and I wouldn't expect you to be able to help on this anymore than I could help myself. I'm pretty adept at solving computer issues on my own usually unless it is something that is locked down by IT. Whatever the issue, its a major pain and I expect I'll still be sorting out the headache into the middle of next week.
For the most part, these machine replacement/refreshers are pretty routine. I'm in State Govt and and mine was just another mandatory 3-year replacement that is handled by NTT. They do this almost daily for the employees throughout the various Departments.

PSOVLSK

Jstas wrote: »

I told you that because, in 2010, the Chief Technical Officer of McAfee Corporation was a guy named George Kurtz.

The Founder and CEO of Crowdstrike is also named George Kurtz.

That's because they are the same guy.

https://en.wikipedia.org/wiki/George_Kurtz

With such a common name as George Kurtz can we really be sure it’s the same guy🤣🤣

Jstas

Wikipedia said it was true