Programming

I work in networking, a job that traditionally has been managed by a terminal and vendor-specific syntax. I used to hate the thought of automation when I was younger, why would something as important as networking be automated? I've made my career on being the clutch guy, troubleshooting complex problems, I love the art of understanding every cog in the machine and being able to visualize it. Then I started learning Python, and learning it was extremely difficult for me. It felt like an eternity between the time I poured myself into learning Python until the time I could actually make things people would want to use.

I was a supervisor working in a NOC. A NOC that had many beaurocratic requirements which got in the way of break/fix operational support, such as having to manually write an email to every customer that had an alarm, and calling every point of contact for that customer, as well as notifying the field techs of outages in their areas, and managing real operational issues. So many times I had to let real work slip through my hands because there were so many calls, so many cases, so many things to do.

Like most NOCs, we viewed alarms from SNMP. When something failed to ping, it would generate a loss of comms alarm. I had this idea to automatically notify the field tech for the specified area when a customer site was downed for more than 30 minutes, and that was a very complex thing to do, it required that I clean a lot of data... I spent days converting things like date strings into proper formatting. Once I presented it, I was told that we couldn't do this, because some political agreement made it to where the NOC was required to provide "positive contact" to other groups. I wrote it, tested it as proof of concept, specd out costs(MRC for the API I was using to send text messages was extremely cheap, it would cost the company about 6 dollars per month). Just like that, it was dead.

My director then wanted me to do something similar for our phone systems. Since our queue depended on user agent availability(your presence status), my boss wanted me to write a program to notify him if someone was unavailable for too long and the reason why. Yes, he wanted to know if someone took more than a few minutes to take a shit or get coffee.

That's when I learned, boomers only care about micro management, not efficiency.

I wouldn’t quite classify this as automation, but I’ve been fighting for the past year for better scripting tools. I work on kiosk-style systems on customer networks. A big part of my job involves connecting to a device, pulling some logs, and running connectivity tests. I created a PowerShell script to automate this and submitted a KB so that others could use it, which sat in the approval queue for a few months before it got rejected.

I reached out to the team who rejected it and was told that all scripts need to be approved by a senior. I told them that a senior had reviewed it and approved it, and linked them the approval which they would have seen anyways. They then said that it also needed approval by the development team. “Okay,” I said. “What’s the process to get that approval? I don’t see any documentation about it.” After a number of emails to several different departments, I found that there is no process. I bugged everyone I could think of but got no replies; my manager got about the same.

In the 12+ months it took to come to that conclusion, I’ve made scripts to automate just about every common fix we apply. Right now most of our KBs instruct us to schedule downtime with the customer to fix things using the GUI, but that’s not necessary for 90% if these issues. I’ve submitted KB revisions for each of these, all of which have been rejected because they need an approval that doesn’t exist.

I’ve brought this up to my manager several times and gotten my seniors to back me up on how much time these scripts save. I’ve shown how effective these scripts are when we have system-wide critical issues where I save us hundreds of man hours of work. None of this has made any difference; apparently the development team just can’t be bothered to create a webform or whatever or even just answer emails.

Lol I love the term distributed monolith. It seems paradoxical but very much a thing I have experienced. I don't think I've ever been rejected when trying to automate something though, that seems crazy.

When the effort to maintain the automation outstrips the benefit. This can happen often in dynamic environments with many moving parts.