For the record: You just ordered me to cause a expensive outage
An electrical engineer faced consequences after removing crucial wiring despite warnings, leading to financial losses. The incident underscores challenges techies encounter with unreasonable demands and poor management.
Read original articleNorman, an electrical engineer, was asked to remove old wiring in a building in London's financial district. Despite Norman's warning that three white cables were crucial for voice and data, his boss insisted on their removal. After complying, traders on upper floors lost all connections, likely resulting in financial losses. Norman, feeling mistreated, left without fixing the issue, a departure from his usual helpful behavior. The incident, shared in The Register's On Call column, highlights the challenges techies face when dealing with unreasonable demands and disrespectful behavior from superiors. This story serves as a cautionary tale about the consequences of ignoring expert advice in technical matters and the impact of poor communication and management in the workplace.
We were having elevated error rates due to an earlier outage, and had just run the next closest hail mary to recovery the switch by telling it to resync all temporary state. The problem was that resync didn't work the way anyone on the team thought it did and the director didn't give enough time for it to be effective. We later found out the resync for a particular record didn't happen until the cell phone itself checked in with the network for a periodic update, so we had to wait for however long that timer was to expire on all cell phones. Luckily there was an active 911 call on the switch which caused the team to delay long enough for me to get the right data (the real world failure rate was far below 1% and starting to trend towards normal after the resync command) and talk the director out of the reboot.
The funny thing is that equipment wasn't even mine, I worked on the internet side of cell phones, and I don't even remember why I was pulled onto that outage call since it didn't have anything to do with my part of the network.
About 30 minutes later, my boss comes into my office and says the owner of the other company called to complain that they felt like "marcus0x62 never wants to help us". (I had - just the week before - gone into the data center on Friday night on about 1 hour notice to do something for them without complaint of any kind.) My boss said "I don't need [the other company] thinking we don't want to help them." He seemed pretty angry and wasn't interested in any explanations. Ok, I said. No problem. I'll take care of it right away. I figured there was, based on past history, about a 25% chance this router would reboot on the spot when I put that card it.
So, I went into the data center and inserted their card into an open slot. And all the blinkenlights lit up. I went back into my office, and the boss was still there. I told him about 25% of our WAN customers would be down for 5 minutes while this router rebooted, and the tech support staff should expect calls.
0 - The router, a Cisco 7200 series device, supported "Online Insertion and Removal", but it didn't always work. And when it didn't work, the router usually rebooted spontaneously.
The system continued to run for about 9 months after I quit before it started having weekly outages, and my new job pays me more with better coworkers.
With holding your labor can be very gratifying.
I had a different but similar situation, but for me, my immediate manager supported me. So I did not have to do what I was being told (I was going to quit). It was a VP who wanted that activity done. All turned out fine for us and I stayed there for many more years.
I remember feeling physically sick, the first time.
By the time I walked out on my last tech* position, it wasn't even a conscious decision; I was gathering my tools and heading to the door without even thinking about it.
* I was 2 hours late on site - after rescuing 2 girls from a wreck on the M5, and being made to hang around and give a statement by the Police; even though I stayed onsite to finish the job beyond normal end of day - I got docked half a day's pay and told I should have left the girls to die.
Fun Fact: EVERY firm I walked out on, went bust within 24 months of my leaving; am I a jinx, or have a 6th sense?