(With all due respect. Please forgive the 15 second preceding commercial)
So, Friday started with me flooding the Internet with unwanted spam. As of yet I have not worked up the cojones to attempt to turn on that faucet again.
While I do offer Internet services this comes at a price. I maintain a business-calibre level of data throughput. My ISP has an average priced package for this. It is “Average” in my opinion insofar as considering the cost of service compared with the speed of the connection compared with the quality of service and support. Actually, putting it down that way may make me have to reconsider my rating.
Part of this package includes a strip of static IP addresses. Most people with Internet get dynamic addresses. What this means is that you have whatever address you are allocated and that’s fine. The reason for Static is because you need a permanent address that people can get to. This is why you can get to Google or Yahoo, but people can’t necessarily find you when you’re on a chat client.
One of the problems that I’ve been suffering is slow download speed and intermittent download “drop-out.” This sounds horrid for my customers, but their download is my upload. My upload has always worked as good if not better than advertised. This download issue has basically been presenting itself as getting 40-60% the download speed I’m paying for.
I had a tech come out early last week. He was running late. No one called to tell me. This was a bit of an annoyance. I do run my business out of my house; so it’s not like my IT department is meeting with the CEO randomly. It does however mean that 3pm – 5pm cuts into my toddler’s dinner when you show up an hour late to start. (Without calling)
45 minutes later. The line looks okay but the speed problem must be internal. “Shrug” says the tech and shuffles off. On Friday another tech shows up to swap out my DSL modem. Normally this will only create about 10 minutes of downtime. He says it might be 15 if the tech in the home office in Kentucky doesn’t do his job quickly.
Actually, the tech in Kentucky does the job in 5 minutes. Of course now my internal network has to be reset. This of course is my bailiwick so I send the tech on his way. Now, the nifty part of this is that my publically served sites are on external static IP addresses. Anything on my internal network shouldn’t affect them.
This is about the time my wife tells me that the internal network has failed. I explain that I have to reset it. Thirty minutes later it’s not working. Internally, I have a DNS server. The internal network is effectively one of my static addresses served via secure WiFi via NAT. For some reason, the WiFi Network has stopped talking to the DNS server altogether. I can ping the server. I can even do NSLOOKUP on the server. I just can’t get the network to do the resolving.
This situation also inconveniences my wife who’s trying to get information and files from our server for her web site and for an art project she’s trying to muster the energy to do for a colleague. The network is frustrating her and in turn really p*£&ing me off! I’m getting short of patience because I just don’t know why things aren’t working.
This disaster proceeds to burn the better part of 3 hours. I reset all the WiFi devices at least 10 times. I reboot the server (3 minute down times) at least 3 times. I finally change all the internal IP addresses and in doing so, make a bizarre mistake. I let the main WiFi unit choose it’s own address. DNS starts moving. But there’s a problem. Now the web server won’t come back up. Ah, there’s a conflict. The WiFi has decided it WANTS one of my served domains. (My wife’s gallery and blog)… Fine… I give her domain the old WiFi Address. Web Server is coming up. DNS is working. Another hour shot to hell.
It is now about 3 pm. I still have no clue what went wrong. I put my wife’s IP address into the server but that takes at least 30 minutes to propagate for me. I go to see if it’s propagated and I can’t hit the site. The network refuses to let me go to the server. (This is a situation we refer to as ITWTF) Then it hits me. I ping the domain’s IP address (this is effectively a “Knock knock…anyone home?”) The ping cannot go to the address. This address does not resolve.
Oh for heaven’s sake. I proceed to ping every one of my Static IPs. Of the n addresses I control. I can ping the first n/2 – 1. So, if I had for example 25, I can now reach 11. The mis-provisioned my account and dropped half of my addresses. I’ve killed the past 4 hours diagnosing a problem that one idiot tech screwed up by transposing ONE DIGIT.
The saving grace is that I prioritize my sites. The paying customers are on the lowest (smallest) IP cluster. As a result, those servers never even blinked at the networking mis-configuration. Some of my personal and favour-domains… those got knocked off for the better part of 4 hrs. I called up my ISP’s ‘elevated support center.’
I have access to the elevated support center because I have a problem that’s been open for the better part of 2 months and I mouthed off on Twitter about the problems I’d been having. When ESC (an ironic acronym) heard about the mistake and the amount of time both my wife and I lost… they were QUICK to fix it. They also offered me a token credit on the bill for the month.
Since then I have put all the IP addresses back where they are supposed to be. My internal network is still ‘renumbered’ because that wasn’t the problem and I just don’t want to put it back again yet. (Maybe later in the week) Less-Than-Humourously, the original problem still exists.
On the upside… on Saturday and Sunday despite this entire disaster happening, I picked up a new hosting customer from GoDaddy who’s been VERY happy with the level of service since joining my server.
Hopefully, I can bring in more paying hosting clients while I’m at it.
I don’t talk about my blood family very often. I’m not just referring to online. My first cousin who is just a year younger than myself, lost her husband suddenly a little over a month ago. CPT Samson T. Luke was on a National Guard drill weekend. The family was notified that his death was an “active” status death. Two weeks later this was changed with no explanation to “inactive” death status. As a result much of the monetary, education, and health care, and funereal benefits are being denied that were previously promised. You can see more on this story from an excerpt of a CNN report.