what happend (FC down part of Saturday & yesterday)?

BIG GRIN!!!!!!!!!!!!!! DROOL DROOL!

 

Good job going to the root of the problem, Rob.
Folks - this sort of thing has happened to many a type of database/server/administrator. I'm on another site that is run by one of the brightest, experienced, computer, internet, technically savvy people on Earth. Absolutely brilliant. Site still crashes 4 times a year.

 

Rob,

You might want to consider running some monitors (or have your hosting company) like BMC or Sitescope - that can check the heart beat of the DB and HW - plus give you the heads up / page/call if there is an issue.

The other consideration - down the road as you gain more volume - will be to upgrade to Sun or Linux for the DB engine and use Oracle... (not that I am bashing MS or anything). Volume becomes a consideration once you hit a certain point. Also utilizing Oracle you also get a better toolset with DBtools, etc.

Just a thought - from one of your F-chat Consultants

 

Oh, Toto, I have a feeling Rob's not in Kansas anymore.

BTW Thanks for fixing it quickly

 

What version of mysql are you running? It should be fairly solid, but AFAIK mysql still dosen't run truely joined queries, one of the reasons I do not like it. Postgre does, and it is also freeware.


Oracle is great if you're making (real) money with the system. their business practices and CEO leave much to be desired however. Prices they charge are highway robbery.

RAID will *SOLVE* your disk problems. Highly recomended (get a standalone unit and bootable FC <qlogic> card for the machine.)

 

Aw, shoot. I guess you didn't see the email about how I had 5 million tax-free US dollars available for you only between the hours of 4 and 6 pm that very day.

 

Maybe this will help next time:

http://www.coopergriggs.com/flash/nf.html

 

Rob- this is just so you boot the machine off the RAID array as well. No sense having a great redundant data drive, only to be screwed when your boot disk goes out.

Solution: just boot off the RAID

 

Finally, something I can contribute

Essentially, there are a few ways to go about this, but knowing how colo facilities charge and the volume you're looking at, there's really only one way to go.

RAID is essential. You'll want to go RAID 5E (RAID with hot spare disk) for the database, and mirrored boot disks. Since your bandwidth and page-per-second serving requirements are relatively light, you could consider a single RAID5E logical drive for both booting and DB, especially with RedHat and MySQL. This will keep your costs low and colo profile reasonable. Personally, I'd go with an IBM x345 or HP DL380 (avoid Dell at all costs!). This would give you the added benefit of more disks spinning (equals faster seek times in DBs) and greater fault tolerance / redundancy. If you use 15K RPM drives, you'll allow yourself plenty of headroom for growth as well.

Multiprocessor won't do anything for you on an Intel platform, because no Wintel/Lintel server on the market today can dynamically disable a processor without going down for a restart. You can beat that by implementing a VMWare virtualization effort, but that is undo cost and complication for your scenario.

Some technologies from HP / IBM do offer fault tolerance at the RAM level beyond the traditional ECC you'd find 3 years ago (essentially these allow you to perform RAID with your RAM). Honestly, you'll have memory failures at least 100X more often than processors, so this is a worthy investment.

Redundant power is a great thing as well, and in server-dense farms like colos, you'll also want redundant fans to cope with what are often poorly ventilated, over-extended server rooms. In these environments, a simple fan failure can lead to multiple failed components.

Rob, I could rant and rave about these things all day. This is what I do, sometimes just for fun If you have any questions, let me know by PM, I'd be happy to help.

 

The above posts are on target... Not sure what Iron you are running on... If it were Blue - I would be going with the IBM 365's rack mounted (Every Maker Compaq/Dell, etc have equilvalant.- you will get your redunant power, etc. Stick with Raid 5 - hot swappable - Disk is cheap - so shouldn't cost you a ton. As mentioned mulitprocessor on Intel will not buy you much - but I would go with a 2 or 4 proc - you will get some basic peroformance gains over a 1 proc. You are still pretty small as far as DB size - 20Gb - SQL will still handle this - Oracle 9i will be the way to go once you double/triple.. I am surprised that your hosting company does not have you on EMC for the DB - like Clarion or Symmetrix (sp) - That way you are utilizing high power disk resource for your DB - and have the Web/App tier run local disk and DB on EMC or Sharkbites, etc. (however this is a higher end solution - may not be an option with the hosting provider).

Ok that is enough geek talk - PM me or give me a call if you need any more suggestions.

Rich

 

Rich is right on with his analysis. Given the opportunity, 4-way Xeon attached to a SAN is the way to go but it is EXPENSIVE. I recently did a "low-end" implementation like this and it ran to about $78K in hardware alone (using M$ software didn't help the costs either ). For maximum scalability I'm completely behind Rich on this one, but the price and requirements don't bear out the expense. You'll see huge improvements in performance just by moving to a RAID5 config (much faster DB lookups) and SCSI (sounds like your provider is serving your pages off a desktop motherboard and hard drive stuffed into a 1U rack mount case, which is very common). Xeons have hyperthreading, which basically makes one processor work like 2. You can offload your Apache onto the strong portion of the CPU and relegate the disk intensive database work to the remaining thread. I've found this to be very effective on sites with larger requirements.

I think the best middle ground for speed and redundancy might be to go with a dual Xeon 1U server (pizza box for all you other geeks out there) attached to a dual ESM external SCSI array, like the IBM EXP400 (I'm a fan of Big Blue). That way, you'd get the benefits of lots of disks (up to 14 platters) for fast database actions and give yourself some nice headroom because the EXP400 would allow you to attach a second server later on and do a cluster. Lower up front costs than SAN, and plenty of expandability in the future. Good solution.

There is also the ebay route. Scoop up a 4-way xeon back off-lease from one of the big ASPs and run with that. Total cost will be less than the solution I proposed, and with some upgrades and tweaks, you'll have more performance than you ever dreamed you'd need.

One more thing. Definitely colo or managed dedicated hosting on YOUR server. There is no substitute as far as I'm concerned.

 

I would like to take this time to apologize for my denial of service attack on Lambochat, which ironically points to...

I will say that it was most likely a mysql fart. It is free, but it is a ram hog.

I might be able to loan a 2way 2ghz xeon box with a gb of ram and raid5 based out of USC (minimal hops to anyplace on earth) and plug in some space on our SQL server. Perl is running on the box, so is compression to nip that transfer size in the butt.

One could argue Unix vs Intel like Ferrari vs Lambo (in no specific order), point is stick with what you are comfy with.

My suggestion would be to allow img code so you are not jamming your self up with image transfers as your b/w hog.

 

Rob - I would lease don't buy - that way you can get out and upgrade sooner as needed - and once you run power through the box - it is not worth much... But I would tell them what you want to lease - or lease your own and co-lo.

Rich

 

rob i have a 4u server here that can handle dual 550 p3's with onboard raid, i'll donate a few scsi drive to the cause. maybe if nothing else it can help handle some of the load by running just the php and the other box can do the db.

I have an ibm drvs 9gb drive that is really fast just for the os, and two 18.4 gb cheetahs, a few ide drives, no ram or cpus, but those are cheap. the motherboard is an intel 440glx+

 

Instead of one very fast server why not two fast servers? Maybe something along the lines of two dual AMD's, one for mail, pictures, whatever and the other for the main F-Chat itself?

I'm trying to get another major forum to do something like this and optimize a lot of their forum stuff because it's just hogged down. vBull isn't the nicest thing once you start hitting a large amount of members. It always needs work.

 

No kidding . The scenario I mentioned should be less than $700 with bandwidth. Certainly no more than $1000. $1000 per month should carry you to 20,000 users realistically.

Have a great weekend!