Ok folks I know you're all dying to know what I've been up to. And I know you're sick and tired of the site being inaccessible, down, slow, etc. So here's what I've been up to.
First, why was the site so unreliable recently? A combination of factors, starting roughly 3 months ago:
- the ISP I was hosted at renumbered their IP network, leaving me without any valid DNS servers for a few days
- the past 2 weeks or so the ISP had been having problems with their T-1 (for some reason, every day starting around 10:00 am). (This has been resolved now.)
- The server the site was hosted on is suffering from uncorrectable ECC RAM errors from time to time resulting in the occasional reboot (not really noticable to a transaction based service like the web, though)
- Thursday night the server lost one of its two mirrored hard drives and needed to be powercycled to reset the SCSI bus. It is still operating with only 1 of its 2 mirrors. It was down for around 15 hours until someone could get to the facility and reset it for me.
- Due to the bad RAM (or motherboard, not sure) and the newly failed disk, this machine is short lived.
Behind the scenes I'd been working on a server upgrade for the site. A SubTalk poster, who I won't name but if he wants to chime in pls feel free, donated to the cause a Sparc 20 and some parts (essentially an identical machine to what we have now, but with a little more RAM and a slightly faster CPU). I also recently purchased a Dell Poweredge 350 rackmount server unit for the purposes of hosting the site.
| | Old Sun Sparc 20 | New Sun Sparc 20 | New Dell
|
|---|
| Disk | 9 gig (2x9 mirrored) | 9 gig (2x9 mirrored) | 10 gig (2x10 mirrored)
|
| Memory | 272 meg | 480 meg | 384 meg
|
| CPU | 2x60 Mhz Sparc | 1 60 Mhz, 1 75 Mhz Sparc | Celeron 600
|
I was also looking around for a new host closer to home that might prove to either be more reliable or allow me easier access to the boxes in case of hardware failure. K2NESoft served us well for many years but was nearly two hours from my home. The aforementioned individual who donated the new Sun hooked me up TTSG Internet Services who will be hosting us (my hardware, their network and facilities) from now on. They have much better bandwidth and multiple peering arrangements so network connectivity will be better.
So, mostly due to the newly failed disk in the main server, last night (Friday) myself and the aforementioned individual scrambled to deploy the two servers, the new Sun and the Dell, at TTSG.
Subtalk and Bustalk are being run on the Dell in their own "virtual hosts", which means if I ever need to move them again we won't have to change the URL's yet again. It also moved the Subtalk and Bustalk posts out of the main tree of www.nycsubway.org documents which means I can keep that tree in sync with my home machine much more easily. Contrary to what I might have led some people to believe last night, this wasn't a software fix; I threw hardware at the problem this time.
The rest of the site is being hosted by BOTH machines together in a round-robin fashion. This will mean that each machine will serve only half the load the former single server needed to deal with.
There's going to be some more bugs to work out as I migrate the rest of the services off the old machine (mail, personal web, etc). So bear with me.
I can't even begin to describe the amount of time, effort, and $$$ required to keep this site up and running. I want to thank the "aforementioned individual", TTSG Internet Services, and K2nesoft for having supported it so far.
-Dave
