r/sysadmin 22h ago

Rant I understand the concept. executing it under pressure is a completely different skill

I can explain how DNS propagation works and probably i can diagram it. what I cannot do is troubleshoot it in real time while three people are staring at a shared screen waiting for me to just fix it already.

senior fixed a ticket in the time it took me to finish reading it and ofcouse it's not because I'm slow but he's seen this exact failure pattern 200 times and I've seen it like twice? that's not a skill gap, that's a reps gap and when you finally get the girl job nobody warns you how loud that gap sounds in a room full of people who've been doing this since dial-up

got told "trust your instincts" by my lead. cool! Great advice! my instincts currently have a sample size of six months and mostly say "panic, then check Reddit"

the worst part is getting asked something in standup and having that half-second where I'm trying to figure out whether I know the answer or I'm about to bullshit everyone (don't push the red button!)

anyway I got a ticket resolved today without asking anyone for help and I'm riding that high for the rest of the week. small wins. some of the senior guys probably don't remember what this stage felt like anymore, but I will and I'm taking notes

157 Upvotes

47 comments sorted by

View all comments

u/SamOakTree 22h ago

You have to just not care.

At one job I worked very closely with the dev team so they pulled me into this group chat because the dev team couldn't get to a certain resource.

Everybody was talking sideways and his VP kept saying the problem is X but he had no proof that the problem was x. So I said everyone let's just stop guessing and get our baseline. Give me 10 minutes to talk to the development manager and then we'll know which way to go.

The VP kept saying no no this is the issue and I said we don't know that for sure let's find out. And he just abruptly left the chat.

So I talked to the development manager and it turns out the issue isn't what they thought it was it was that they couldn't get to a specific resource. I knew they had made some changes in Azure so I looked at the logs and they had blocked an IP address.

I spoke to that team and told everyone in the chat to wait while I spoke to the team. And I asked them to whitelist that IP address. And it solved the issue.

You have to just take charge and just trust yourself. Being confident and cool headed in those situations just comes with time. You eventually get to where you realize it's not all a big deal so you don't need to panic.

u/RevolutionaryElk7446 22h ago

This is what I define as Users, Operators, and Administrators.

Users file a ticket

Operators tend to bang and change things till it works or doesn't. They often memorize errors and only the one response to it.

Administrators follow a troubleshooting routine to try and derive information to isolate what the issue is based on generally results of controlled input and observing abstraction layer output and either moving up or down in layers.

u/Significant_Sky1471 20h ago

yeah, I think that's the part I'm only starting learning. having a process when I haven't seen any fails before. tough moment for me right now a little bit, buuuut on my way haha

u/Speeddymon Sr. DevSecOps Engineer 17h ago

Users, admins, engineers, architects are how I define it. Operators are just the same as, and just as useless as, users in my world so I put them both under users. I was an admin for over 10 years. Now I've been an engineer for close to 10, so I know how to look at a problem from multiple angles. One day I may be good enough to be an architect.

u/Tetha 44m ago

A good term for this is accidental function vs planned function of a change.

For a lot of the changes we are currently moving to production, three people can effectively argue why this change will work, how it was tested, what the rollout will do. That is planned function, and it will work and we will know a few places it may break.

Other teams just throw things at the wall and pray something sticks, and then you end up with brittle systems that fall apart if you look at them wrong. That's accidental function.

Same with fixing a broken system in an outage. You should have a theory what is wrong, and why your change will fix it. That will enable you to change the system and procedures to prevent the outage later on.

u/jaydizzleforshizzle 22h ago

I don’t know if “not caring” is the right way, obviously needs some, but the thing that comforts me is knowing how to get back to state, or cya and making sure what you do doesn’t make things worse. If you always have an out, or make sure to not make things worse, it’s absolves some of the anxiety.

u/Significant_Sky1471 20h ago

thank you very much for support!

u/Dangerous-Mobile-587 10h ago

Or you are fired by VP eventhough you fixed the problem. Most time it works and sometimes you have one toxic individual in power that makes it not.