June 29th, 2024

For the record: You just ordered me to cause a expensive outage

An electrical engineer faced consequences after removing crucial wiring despite warnings, leading to financial losses. The incident underscores challenges techies encounter with unreasonable demands and poor management.

Read original articleLink Icon
For the record: You just ordered me to cause a expensive outage

Norman, an electrical engineer, was asked to remove old wiring in a building in London's financial district. Despite Norman's warning that three white cables were crucial for voice and data, his boss insisted on their removal. After complying, traders on upper floors lost all connections, likely resulting in financial losses. Norman, feeling mistreated, left without fixing the issue, a departure from his usual helpful behavior. The incident, shared in The Register's On Call column, highlights the challenges techies face when dealing with unreasonable demands and disrespectful behavior from superiors. This story serves as a cautionary tale about the consequences of ignoring expert advice in technical matters and the impact of poor communication and management in the workplace.

Link Icon 10 comments
By @kevin_nisbet - 5 months
Nothing like this story of getting bullied to do something, but I once talked a director out of rebooting the cell phone switch that handled all cellular phone calls for I think it was 4 provinces in Canada. This is something I don't think we had ever done before, and was expected to be a 100% outage for all cell phones calls + 911 for like 20 - 25 minutes or something.

We were having elevated error rates due to an earlier outage, and had just run the next closest hail mary to recovery the switch by telling it to resync all temporary state. The problem was that resync didn't work the way anyone on the team thought it did and the director didn't give enough time for it to be effective. We later found out the resync for a particular record didn't happen until the cell phone itself checked in with the network for a periodic update, so we had to wait for however long that timer was to expire on all cell phones. Luckily there was an active 911 call on the switch which caused the team to delay long enough for me to get the right data (the real world failure rate was far below 1% and starting to trend towards normal after the resync command) and talk the director out of the reboot.

The funny thing is that equipment wasn't even mine, I worked on the internet side of cell phones, and I don't even remember why I was pulled onto that outage call since it didn't have anything to do with my part of the network.

By @marcus0x62 - 5 months
A long time ago a customer called in and asked me to install some hardware they sent (an Ethernet card for a router) right away. I declined - the router it needed to be installed in was a shared device, and we'd had problems with hot swapping cards[0] in this model of router before. I told them I'd do it after hours that evening, but not during the day. This would have been maybe a 3 or 4 hour delay for them.

About 30 minutes later, my boss comes into my office and says the owner of the other company called to complain that they felt like "marcus0x62 never wants to help us". (I had - just the week before - gone into the data center on Friday night on about 1 hour notice to do something for them without complaint of any kind.) My boss said "I don't need [the other company] thinking we don't want to help them." He seemed pretty angry and wasn't interested in any explanations. Ok, I said. No problem. I'll take care of it right away. I figured there was, based on past history, about a 25% chance this router would reboot on the spot when I put that card it.

So, I went into the data center and inserted their card into an open slot. And all the blinkenlights lit up. I went back into my office, and the boss was still there. I told him about 25% of our WAN customers would be down for 5 minutes while this router rebooted, and the tech support staff should expect calls.

0 - The router, a Cisco 7200 series device, supported "Online Insertion and Removal", but it didn't always work. And when it didn't work, the router usually rebooted spontaneously.

By @hiddencost - 5 months
I once fought a bad boss for a year. I finally recognized my only remaining power was my ability to quit.

The system continued to run for about 9 months after I quit before it started having weekly outages, and my new job pays me more with better coworkers.

With holding your labor can be very gratifying.

By @jmclnx - 5 months
I wonder how it turned out for Norman, I hope his manager did not blame him. These days seems you need provable evidence to be safe in some jobs.

I had a different but similar situation, but for me, my immediate manager supported me. So I did not have to do what I was being told (I was going to quit). It was a VP who wanted that activity done. All turned out fine for us and I stayed there for many more years.

By @CHB0403085482 - 5 months
The more often you walk out on an abusive boss, the easier it gets.

I remember feeling physically sick, the first time.

By the time I walked out on my last tech* position, it wasn't even a conscious decision; I was gathering my tools and heading to the door without even thinking about it.

* I was 2 hours late on site - after rescuing 2 girls from a wreck on the M5, and being made to hang around and give a statement by the Police; even though I stayed onsite to finish the job beyond normal end of day - I got docked half a day's pay and told I should have left the girls to die.

Fun Fact: EVERY firm I walked out on, went bust within 24 months of my leaving; am I a jinx, or have a 6th sense?

By @malfist - 5 months
The article just feels like rage bait. No details to confirm it's veracity and plenty of it sounds made up petty revenge stories.
By @abeppu - 5 months
Does an "electrical engineer" have a different job description in the UK than in the US?
By @darksim905 - 5 months
Who writes like this. Good grief.