Moving a Multi-Million dollar business’ data to a new infrastructure (all by myself…) GULP

TL;DR: Upgrade took a while but went really smooth. Ran into several small events but it all worked out and now on a more stable platform that supports even more failures without a hiccup.

Over the past few weeks I was tasked with rebuilding our internal IT infrastructure, as our old server originally built in 2008-2009 was showing it’s age. Although at the end it was a beastly system featuring dual X5460 processors it started struggling with our automation tasks that keep all of our internal systems running. cronjob and task scheduler (windows) are your friend!

I was telling our owners and managers that we needed to upgrade our main server really soon as it was getting to the point of it being unreliable… and wouldn’t you know it… PROBLEMS!

The existing machine was a HP Proliant ML350 G5 and it featured a very nice Raid system (HP E200i Smart Array). It has helped us out of a lot of jams because old hard drives like to bite the dust considering they are worked non-stop 24/7. Unfortunately this error message was something I haven’t seen before. It stated it was ready for rebuild but no matter what I would do it would refuse to start.

Fighting with it during boot up to force it to start recovery it would fail after a minute and go back to operating poorly and locking up.

What route do we take?

There are a lot of routes to take going forward. I never want to buy new servers because we do not need necessarily cutting edge servers (plus you pay way too much for nothing when it comes to new servers!)

After looking at options I placed two options in front of the company… We buy a newer (used) server again, or we build our own mini rack. After thinking it through I went with the custom market.

I know darn well that current generation hardware is pretty reliable, and considering I replaced my lan party room with new 2600 Ryzen computers, thought it could be adapted to a server environment.

Instead of one server for $1,600 (used), I went with building two exactly identical systems with AMD Ryzen 2600 processors, and 32GB of DDR4 ram. There are two RAID 1 arrays in each, one for SSDs and one for HDDs. I was able to build two of those systems and a nice wall enclosure for the systems for the same price!

Virtualization!

The one thing I wanted to do on the new system versus the old was build it all on top of Virtualization. I haven’t had to work with any other Virtualized environment than VirtualBox, but for a server environment I wanted something a bit more reliable and robust (at least from the looks of it), so I went Hyper-V 2016 Server (Core) for both. I did have 2019 lined up but they had issues with supporting our Raid Controller on both machines and 2016 worked out just fine!

For those who do not know, there are big benefits to building servers on top of a virtualized platform… While you may lose a bit of performance to the ’emulation’ layers of things like network cards, and other I/O devices, you have easy recovery and other nice features.

For example, if our old server died, I had other hardware to load our snapshots/backups onto, but if the hardware is different enough, windows will not easily be able to boot up properly without a lot of fighting and configuring. In a virtualized environment you can easily drop and build a matched system and it will boot and operate without a hitch! This is what I wanted, and since both servers are identical it makes that that much easier allowing one server to take over for the other in case of a hardware failure.

Splitting the Work

It made sense to split the work up a bit. Our old machine was a domain controller, file server, web server, database server, quickbooks server and more! Everything was on one point of failure. I hate that.

The new set up had one machine handle the web server and algorithms, and one handled file server, domain controller, and quickbooks. It was a good divide.

Improving IIS

Upon initial set up of the IIS web server I noticed scripts were taking longer to computer than our old server. I was baffled as the drives were much quicker, the ram was much faster (DDR4 VS DDR2) and each server had 12 threads @ 3.6ghz. Reading up on everything I went through and tweaked PHP, MariaDB, and IIS. And after much work got things running lightning quick. The important improvement being adjusting how many instances of the FastCGI PHP module were allowed. Moving it from 0 (unlimited) to 24 helped tremendously. For some reason it would never make more than 1 instance no matter what. (IIS 10)

We have room to grow!

With the new machines running smoothly for a week now I finally took some ‘free’ vacation due to all the extra hours I pulled. This is the type of stuff I live for. We have more power than any of our competitors, and we are soon going to be adding and adding more AI experiences for our customers, having our system analyze our customers and tailor options to them they may want.

I am excited for what the future brings for these beautiful machines. Not to mention they are named after the HBO show, “Silicon Valley”.

Oh yeah, and we do backups!

For those thinking “Wow you’re stupid for trusting residential components”, we do three levels of backups. Snapshot backups handled locally, also file level backup done locally AND offsite. Yes, backups are checked and tested.

PHP Generate Color in Rainbow

I had to create a script that generated rainbow between 2 to 768 colors in the rainbow. It returns an array that can be referenced. It’s ideal for 5 to 700 results.

<?PHP
// Generate Rainbow Points.
function generatecolorcodes($numberofcolor)
{
	// Find numbers to pick.
	if ($numberofcolor > 0 && $numberofcolor <= 768)
	{
		$divis = round(768 / $numberofcolor);
		$s = 0;
		while ($s < $numberofcolor)
		{
			$cwheel = $s * $divis;
			if ($cwheel >= 0 && $cwheel < 256)
			{
				$red = 255 - $cwheel;
				$green = 0 + $cwheel;
				$blue = 0;
			}
			if ($cwheel >= 256 && $cwheel < 513)
			{
				$red = 0;
				$green = 255 - ($cwheel - 255);
				$blue = 0 + ($cwheel - 255);
			}
			if ($cwheel >= 513 && $cwheel < 768)
			{
				$red = 0 + ($cwheel - 512);
				$green = 0;
				$blue = 255 - ($cwheel - 512);
			}
			$red = dechex($red);
			$green = dechex($green);
			$blue = dechex($blue);
			if (strlen($blue)<2) {$blue = '0'.$blue;}
			if (strlen($green)<2){$green = '0'.$green;}
			if (strlen($red)<2){$red = '0'.$red;}
			$outarray[$s] = "#" . $red . $green . $blue;
			$s++;
		}
	}
	return $outarray;
}
?>

Llama Punter Updates

I went through and made some cosmetic updates to Llama Punter (for UT2004) and updated how the feeds look as well as the logo. Check it out at https://punter.vengefulllama.com

Join the Discord

In the past few weeks it has been clear that a lot of people want to get back together and play games. On the weekends we have been playing a lot of Assault in UT2004. Join in the discord (Link at PWC-Gaming.com) and get in on the fun. We’d love to have you there! Right now there is 25 people part of it.

It’s hard to top the feeling.

Today I got the reminder of the wonders of automation. About 2 years ago I rolled out a very powerful system that lets our customers book their appointments online. The website makes it look a lot easier than it took to make it work reliably.

First off, our scheduling system is written using a very old database method (MDB) and it is garbage and will always be garbage. To limit the amount of times we have to talk to it, I built out a caching system that does quick checks every minute and full syncs every 5 minutes. Now we have a copy of the database stored in a much easier to work with format.

Secondly, I had to build an extremely complicated and crazy CPU intensive system that looks at ALL appointments in our system and starts generating all possible open times that we can schedule appointments as well as locations and how much open time each spot would have. I made several code improvements and still it takes about 65 seconds to operate on one a compute node (8 core)

Third, we have latitude and longitude stored in caching tables with the open time cache data so we can triangulate the most efficient times to schedule the appointment. Upon a customer entering their address into the system it will take that information and determine the best results and return a score.

It is hard to top the feeling that you essentially built an employee that is there for your customers 24/7, doing your job and other’s jobs. I feel really euphoric thinking about it, and how quite a few people just today utilized the system.

Self-Service is something I believe should be an option in nearly everything. Customers (no matter the industry) wants the option to do it themselves. Never get rid of customer service options like phone support, however, if you do it right customers will start to utilize the self-service and open up time for your employees to get other work done.

Don’t cheap out on web hosting.

Just heard a horror story of a company who decided to change hosts to their new IT guys and it backfired. It’s not best to trust a small business web hosting company who does not handle web hosting primarily. They may have servers that work but they may not have reliable services like Backups, Backup Power, etc.

Websites are important and many use the hosting account to run their emails too. They did in this case and lost them all. I had them on a service that had backups. The hosts I recommend at this point in time is Dreamhost or Hostgator. I prefer Dreamhost for their free SSL certs (using Let’s Encrypt) plus user isolation incase of security incidents on just one of your sites. If you’re used to using cpanel go with hostgator. Neither will give you 100% uptime but they do their best (much like others I didn’t name) to make sure things are fixed quickly.

Net Neutrality

It’s a shame this is where we are. The FCC is going to be voting to strip Net Neutrality rights. This is not good, and speaking as a person who has set up and manage TONS of servers on different carriers I can tell you this breaks rules that helps keep everyone honest.

While I don’t believe ISPs will be doing tiers in the way people over-react to, I do see legitimate throttling like so…

(These are examples, they are not doing this currently, only theoretical)

Spectrum has their own voice service, to encourage people to use their service they will intentionally starve connections to that service to make their service look better. They could do anything from providing poor routing, not upgrading where needed to handle the load or just having a lot of packet loss making the phone call sound terrible.

AT&T offers DirecTV Now and U-Verse service. They could slow down or not give proper bandwidth to services like Sling TV, PlayStation Vue, Hulu TV, etc. This would make their services look better by giving the illusion that a service is worse when it really wasn’t the producing company’s fault.

Verizon has a NFL deal and if you were trying to watch NFL on their network from a different legit source they could slow you down or just not provide reliable service so you would get a bad feeling about what you’re paying for and dump it for verizon’s.

In the past things like this has happened, and it would make a triumphant return if their lobbyists get their way.

This should not be a divided issue. There is literally no benefit to the customers, only the freedom for the companies to do what they do best… Try to get all the money they can from us pathetic chumps. From what I have seen our current government is not remotely listening to the people at this time, they are setting up their friends and other wealthy donors with the bonuses that they really want. We have to do what we can, call your politicians… It’s pretty much all we can do.

https://www.battleforthenet.com/ Get more information here.

Fraghouse Nations server is back up for the holidays

One of the game servers I set up a decade ago is back for the winter. Updated with TitanRPG for better leveling. Fraghouse Nations is a group of custom mods that weren’t necessarily designed to work together all into one unique experience. It uses standard invasion but with a lot more monsters and on custom maps designed for this mode. There are customized Vehicles, Weapons, Maps, Monsters, and more?! The leveling is set up so you can get to a point where it’s relatively easy within about 45 minutes.

You can find this server under the Assault Tab where you will find the other legacy PWC-Gaming servers.

UT2004 Files Sorted a Bit

Thanks for a fan (GoPostal) I thought it was finally time to go through some of the archives of stuff I have for Unreal Tournament 2004. One of the things I double checked was the LAN Party in a box server for UT2004. As it stood it was functional but when I set it up it was configured to a redirect server I hosted that was not active anymore. Since I have the hosting ability I put the files back up on there, so the server will function 100% out of the box.

I also uploaded to the additional files area the Game Mode Override feature, which is quite helpful if you have a lot of servers but want them to appear all in the same game mode list. That’s what I originally set it up for is to allow our invasion server and freon servers to appear under assault along with all the other servers we had. It’s easy to set up and no bull to configure, just add the mutator and it will tell the master server you’re running that mode no matter what you’re doing.