Moving a Multi-Million dollar business’ data to a new infrastructure (all by myself…) GULP

TL;DR: Upgrade took a while but went really smooth. Ran into several small events but it all worked out and now on a more stable platform that supports even more failures without a hiccup.

Over the past few weeks I was tasked with rebuilding our internal IT infrastructure, as our old server originally built in 2008-2009 was showing it’s age. Although at the end it was a beastly system featuring dual X5460 processors it started struggling with our automation tasks that keep all of our internal systems running. cronjob and task scheduler (windows) are your friend!

I was telling our owners and managers that we needed to upgrade our main server really soon as it was getting to the point of it being unreliable… and wouldn’t you know it… PROBLEMS!

The existing machine was a HP Proliant ML350 G5 and it featured a very nice Raid system (HP E200i Smart Array). It has helped us out of a lot of jams because old hard drives like to bite the dust considering they are worked non-stop 24/7. Unfortunately this error message was something I haven’t seen before. It stated it was ready for rebuild but no matter what I would do it would refuse to start.

Fighting with it during boot up to force it to start recovery it would fail after a minute and go back to operating poorly and locking up.

What route do we take?

There are a lot of routes to take going forward. I never want to buy new servers because we do not need necessarily cutting edge servers (plus you pay way too much for nothing when it comes to new servers!)

After looking at options I placed two options in front of the company… We buy a newer (used) server again, or we build our own mini rack. After thinking it through I went with the custom market.

I know darn well that current generation hardware is pretty reliable, and considering I replaced my lan party room with new 2600 Ryzen computers, thought it could be adapted to a server environment.

Instead of one server for $1,600 (used), I went with building two exactly identical systems with AMD Ryzen 2600 processors, and 32GB of DDR4 ram. There are two RAID 1 arrays in each, one for SSDs and one for HDDs. I was able to build two of those systems and a nice wall enclosure for the systems for the same price!

Virtualization!

The one thing I wanted to do on the new system versus the old was build it all on top of Virtualization. I haven’t had to work with any other Virtualized environment than VirtualBox, but for a server environment I wanted something a bit more reliable and robust (at least from the looks of it), so I went Hyper-V 2016 Server (Core) for both. I did have 2019 lined up but they had issues with supporting our Raid Controller on both machines and 2016 worked out just fine!

For those who do not know, there are big benefits to building servers on top of a virtualized platform… While you may lose a bit of performance to the ’emulation’ layers of things like network cards, and other I/O devices, you have easy recovery and other nice features.

For example, if our old server died, I had other hardware to load our snapshots/backups onto, but if the hardware is different enough, windows will not easily be able to boot up properly without a lot of fighting and configuring. In a virtualized environment you can easily drop and build a matched system and it will boot and operate without a hitch! This is what I wanted, and since both servers are identical it makes that that much easier allowing one server to take over for the other in case of a hardware failure.

Splitting the Work

It made sense to split the work up a bit. Our old machine was a domain controller, file server, web server, database server, quickbooks server and more! Everything was on one point of failure. I hate that.

The new set up had one machine handle the web server and algorithms, and one handled file server, domain controller, and quickbooks. It was a good divide.

Improving IIS

Upon initial set up of the IIS web server I noticed scripts were taking longer to computer than our old server. I was baffled as the drives were much quicker, the ram was much faster (DDR4 VS DDR2) and each server had 12 threads @ 3.6ghz. Reading up on everything I went through and tweaked PHP, MariaDB, and IIS. And after much work got things running lightning quick. The important improvement being adjusting how many instances of the FastCGI PHP module were allowed. Moving it from 0 (unlimited) to 24 helped tremendously. For some reason it would never make more than 1 instance no matter what. (IIS 10)

We have room to grow!

With the new machines running smoothly for a week now I finally took some ‘free’ vacation due to all the extra hours I pulled. This is the type of stuff I live for. We have more power than any of our competitors, and we are soon going to be adding and adding more AI experiences for our customers, having our system analyze our customers and tailor options to them they may want.

I am excited for what the future brings for these beautiful machines. Not to mention they are named after the HBO show, “Silicon Valley”.

Oh yeah, and we do backups!

For those thinking “Wow you’re stupid for trusting residential components”, we do three levels of backups. Snapshot backups handled locally, also file level backup done locally AND offsite. Yes, backups are checked and tested.