this post was submitted on 19 Jul 2024
2 points (100.0% liked)

Technology

59566 readers
4839 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
 

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...

you are viewing a single comment's thread
view the rest of the comments
[–] sasquash@sopuli.xyz 0 points 4 months ago (3 children)

never do updates on a Friday.

[–] spyd3r@sh.itjust.works 0 points 4 months ago (4 children)

Never update unless something is broken.

[–] iknowitwheniseeit@lemmynsfw.com 0 points 4 months ago (1 children)
[–] Nachorella@lemmy.sdf.org 0 points 4 months ago (1 children)

If it was Arch you'd update once every 15 minutes whether anything's broken or not.

[–] sugar_in_your_tea@sh.itjust.works 0 points 4 months ago (2 children)

I use Tumbleweed, so I only get updates once/day, twice if something explodes. I used to use Arch, so my update cycle has lengthened from 1-2x/day to 1-2x/week, which is so much better.

[–] wreckedcarzz@lemmy.world 0 points 4 months ago

gets two update notifications

Ah, must be explosion Wednesday

[–] Nachorella@lemmy.sdf.org 0 points 4 months ago

I really like the tumbleweed method, seems like the best compromise between arch and debian style updates.

[–] Passerby6497@lemmy.world 0 points 4 months ago

That's advice so smart you're guaranteed to have massive security holes.

[–] Toribor@corndog.social 0 points 4 months ago (1 children)

This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.

[–] Ookami38@sh.itjust.works 0 points 4 months ago (1 children)

I think vulnerabilities found count as "something broken" and chap you replied to simply did not think that far ahead hahah

[–] huginn@feddit.it 0 points 4 months ago (2 children)

For real - A cyber security company should basically always be pushing out updates.

Exactly. You don't know what the vulnerabilities are, but the vendors pushing out updates typically do. So stay on top of updates to limit the attack surface.

Major releases can wait, security updates should be pushed as soon as they can be proven to not break prod.

[–] wreckedcarzz@lemmy.world 0 points 4 months ago

always pushing out updates

Notes: Version bump: Eric is a twat so I removed his name from the listed coder team members on the about window.

git push --force

leans back in chair productive day, productive day indeed

[–] Hotzilla@sopuli.xyz 0 points 4 months ago

This is AV, and even possible that it is part of definitions (for example some windows file deleted as false positive). You update those daily.

[–] rozodru@lemmy.ca 0 points 4 months ago (3 children)

yeah someone fucked up here. I mean I know you're joking but I've been in tech for like 20+ years at this point and it was always, always, ALWAYS, drilled into me to never do updates on Friday, never roll anything out to production on Friday. Fridays were generally meant for code reviews, refactoring in test, work on personal projects, raid the company fridge for beer, play CS at the office, whatever just don't push anything live or update anything.

And especially now the work week has slimmed down where no one works on Friday anymore so you 100% don't roll anything out, hell it's getting to the point now where you just don't roll anything out on a Thursday afternoon.

[–] sasquash@sopuli.xyz 0 points 4 months ago

Actually I was not even joking. I also work in IT and have exactly the same opinion. Friday is for easy stuff!

[–] 0x0@lemmy.dbzer0.com 0 points 4 months ago (2 children)

And especially now the work week has slimmed down where no one works on Friday anymore

Excuse me, what now? I didn't get that memo.

[–] meanmon13@lemmy.zip 0 points 4 months ago (1 children)

Yeah it's great :-) 4 10hr shifts and every weekend is a 3 day weekend

[–] jedibob5@lemmy.world 0 points 4 months ago* (last edited 4 months ago) (1 children)

Is the 4x10 really worth the extra day off? Tbh I'm not sure it would work very well for me... I find just one 10-hour day to be kinda draining, so doing that 4 times a week every week feels like it might just cancel out any benefits of the extra day off.

[–] meanmon13@lemmy.zip 0 points 4 months ago

I am very used to it so I don't find it draining. I tried 5x8 once and it felt more like working an extra day than getting more time in the afternoon. If that makes sense. I also start early around 7am, so I am only staying a little later than other people

[–] rozodru@lemmy.ca 0 points 4 months ago

sorry :( yeah I, at most, do 3 days in the office now. Fridays are a day off and Mondays mostly everyone just works from home if at all. downtown Toronto on Mondays and Fridays is pretty much dead.

[–] Blackmist@feddit.uk 0 points 4 months ago (1 children)

Yep, anything done on Friday can enter the world on a Monday.

I don't really have any plans most weekends, but I sure as shit don't plan on spending it fixing Friday's fuckups.

And honestly, anything that can be done Monday is probably better done on Tuesday. Why start off your week by screwing stuff up?

We have a team policy to never do externally facing updates on Fridays, and we generally avoid Mondays as well unless it's urgent. Here's roughly what each day is for:

  • Monday - urgent patches that were ready on Friday; everyone WFH
  • Tuesday - most releases; work in-office
  • Wed - fixing stuff we broke on Tuesday/planning the next release; work in-office
  • Thu - fixing stuff we broke on Tuesday, closing things out for the week; WFH
  • Fri - documentation, reviews, etc; WFH

If things go sideways, we come in on Thu to straighten it out, but that almost never happens.

[–] merc@sh.itjust.works 0 points 4 months ago

You posted this 14 hours ago, which would have made it 4:30 am in Austin, Texas where Cloudstrike is based. You may have felt the effect on Friday, but it's extremely likely that the person who made the change did it late on a Thursday.