Yesterday, I came in to work just like any other day. While checking some bugs out, I went to load the staging version of our site. Lo and behold, the site was down. No big deal, I thought, maybe Apache died and needs to be restarted. I’d been messing with the Apache configs quite heavily last week, and so I just assumed that was what caused the problem.
I brought up my console and typed in the ssh command to which there was no response. The server is located in my building, so I walked over to see what the deal was. The server was still running, so I gave it a restart. Though Linux is known for being rock solid, you’d be amazed at the problems a restart will solve. Well, it finished restarting and still no connection.
After poking around some more, I discovered the server wasn’t even online. I couldn’t ping out. I tried disabling the firewall for awhile, no luck. I tried changing the DNS to OpenDNS, still nothing. Finally, I walked over to the IT admin’s office and asked him if there had been any changes over the weekend. Of course his reply was a “Yes…”
It seems that over the weekend, they decided to switch the IP block they had. And it wasn’t like a subnet changed, the entire block was different. A change like this doesn’t just happen. There needs to be planning done so that server configurations can be changed, DNS needs time to propagate, people need to be notified. Apparently, I wasn’t on that “to notify” list. I was told “we tried to find you on Friday, but we couldn’t.” Bullshit. I was in the office all day on Friday and the IT admin even came in to our office earlier in the day.
I spent the first half of my day yesterday dealing with getting our staging server up and running. All that really needed to be done was to change a few config files. A task I could’ve easily prepared for last weekend and finished in minutes yesterday had I known in advance. That kind of inconsiderate behavior is unprofessional and it really pissed me off.
