SgtMuffin

TL;DR: Upgrade took a while but went really smooth. Ran into several small events but it all worked out and now on a more stable platform that supports even more failures without a hiccup.

Over the past few weeks I was tasked with rebuilding our internal IT infrastructure, as our old server originally built in 2008-2009 was showing it’s age. Although at the end it was a beastly system featuring dual X5460 processors it started struggling with our automation tasks that keep all of our internal systems running. cronjob and task scheduler (windows) are your friend!

I was telling our owners and managers that we needed to upgrade our main server really soon as it was getting to the point of it being unreliable… and wouldn’t you know it… PROBLEMS!

The existing machine was a HP Proliant ML350 G5 and it featured a very nice Raid system (HP E200i Smart Array). It has helped us out of a lot of jams because old hard drives like to bite the dust considering they are worked non-stop 24/7. Unfortunately this error message was something I haven’t seen before. It stated it was ready for rebuild but no matter what I would do it would refuse to start.

Fighting with it during boot up to force it to start recovery it would fail after a minute and go back to operating poorly and locking up.

What route do we take?

There are a lot of routes to take going forward. I never want to buy new servers because we do not need necessarily cutting edge servers (plus you pay way too much for nothing when it comes to new servers!)

After looking at options I placed two options in front of the company… We buy a newer (used) server again, or we build our own mini rack. After thinking it through I went with the custom market.

I know darn well that current generation hardware is pretty reliable, and considering I replaced my lan party room with new 2600 Ryzen computers, thought it could be adapted to a server environment.

Instead of one server for $1,600 (used), I went with building two exactly identical systems with AMD Ryzen 2600 processors, and 32GB of DDR4 ram. There are two RAID 1 arrays in each, one for SSDs and one for HDDs. I was able to build two of those systems and a nice wall enclosure for the systems for the same price!

Virtualization!

The one thing I wanted to do on the new system versus the old was build it all on top of Virtualization. I haven’t had to work with any other Virtualized environment than VirtualBox, but for a server environment I wanted something a bit more reliable and robust (at least from the looks of it), so I went Hyper-V 2016 Server (Core) for both. I did have 2019 lined up but they had issues with supporting our Raid Controller on both machines and 2016 worked out just fine!

For those who do not know, there are big benefits to building servers on top of a virtualized platform… While you may lose a bit of performance to the ’emulation’ layers of things like network cards, and other I/O devices, you have easy recovery and other nice features.

For example, if our old server died, I had other hardware to load our snapshots/backups onto, but if the hardware is different enough, windows will not easily be able to boot up properly without a lot of fighting and configuring. In a virtualized environment you can easily drop and build a matched system and it will boot and operate without a hitch! This is what I wanted, and since both servers are identical it makes that that much easier allowing one server to take over for the other in case of a hardware failure.

Splitting the Work

It made sense to split the work up a bit. Our old machine was a domain controller, file server, web server, database server, quickbooks server and more! Everything was on one point of failure. I hate that.

The new set up had one machine handle the web server and algorithms, and one handled file server, domain controller, and quickbooks. It was a good divide.

Improving IIS

Upon initial set up of the IIS web server I noticed scripts were taking longer to computer than our old server. I was baffled as the drives were much quicker, the ram was much faster (DDR4 VS DDR2) and each server had 12 threads @ 3.6ghz. Reading up on everything I went through and tweaked PHP, MariaDB, and IIS. And after much work got things running lightning quick. The important improvement being adjusting how many instances of the FastCGI PHP module were allowed. Moving it from 0 (unlimited) to 24 helped tremendously. For some reason it would never make more than 1 instance no matter what. (IIS 10)

We have room to grow!

With the new machines running smoothly for a week now I finally took some ‘free’ vacation due to all the extra hours I pulled. This is the type of stuff I live for. We have more power than any of our competitors, and we are soon going to be adding and adding more AI experiences for our customers, having our system analyze our customers and tailor options to them they may want.

I am excited for what the future brings for these beautiful machines. Not to mention they are named after the HBO show, “Silicon Valley”.

Oh yeah, and we do backups!

For those thinking “Wow you’re stupid for trusting residential components”, we do three levels of backups. Snapshot backups handled locally, also file level backup done locally AND offsite. Yes, backups are checked and tested.

I had to create a script that generated rainbow between 2 to 768 colors in the rainbow. It returns an array that can be referenced. It’s ideal for 5 to 700 results.

<?PHP
// Generate Rainbow Points.
function generatecolorcodes($numberofcolor)
{
	// Find numbers to pick.
	if ($numberofcolor > 0 && $numberofcolor <= 768)
	{
		$divis = round(768 / $numberofcolor);
		$s = 0;
		while ($s < $numberofcolor)
		{
			$cwheel = $s * $divis;
			if ($cwheel >= 0 && $cwheel < 256)
			{
				$red = 255 - $cwheel;
				$green = 0 + $cwheel;
				$blue = 0;
			}
			if ($cwheel >= 256 && $cwheel < 513)
			{
				$red = 0;
				$green = 255 - ($cwheel - 255);
				$blue = 0 + ($cwheel - 255);
			}
			if ($cwheel >= 513 && $cwheel < 768)
			{
				$red = 0 + ($cwheel - 512);
				$green = 0;
				$blue = 255 - ($cwheel - 512);
			}
			$red = dechex($red);
			$green = dechex($green);
			$blue = dechex($blue);
			if (strlen($blue)<2) {$blue = '0'.$blue;}
			if (strlen($green)<2){$green = '0'.$green;}
			if (strlen($red)<2){$red = '0'.$red;}
			$outarray[$s] = "#" . $red . $green . $blue;
			$s++;
		}
	}
	return $outarray;
}
?>

Just heard a horror story of a company who decided to change hosts to their new IT guys and it backfired. It’s not best to trust a small business web hosting company who does not handle web hosting primarily. They may have servers that work but they may not have reliable services like Backups, Backup Power, etc.

Websites are important and many use the hosting account to run their emails too. They did in this case and lost them all. I had them on a service that had backups. The hosts I recommend at this point in time is Dreamhost or Hostgator. I prefer Dreamhost for their free SSL certs (using Let’s Encrypt) plus user isolation incase of security incidents on just one of your sites. If you’re used to using cpanel go with hostgator. Neither will give you 100% uptime but they do their best (much like others I didn’t name) to make sure things are fixed quickly.

It’s a shame this is where we are. The FCC is going to be voting to strip Net Neutrality rights. This is not good, and speaking as a person who has set up and manage TONS of servers on different carriers I can tell you this breaks rules that helps keep everyone honest.

While I don’t believe ISPs will be doing tiers in the way people over-react to, I do see legitimate throttling like so…

(These are examples, they are not doing this currently, only theoretical)

Spectrum has their own voice service, to encourage people to use their service they will intentionally starve connections to that service to make their service look better. They could do anything from providing poor routing, not upgrading where needed to handle the load or just having a lot of packet loss making the phone call sound terrible.

AT&T offers DirecTV Now and U-Verse service. They could slow down or not give proper bandwidth to services like Sling TV, PlayStation Vue, Hulu TV, etc. This would make their services look better by giving the illusion that a service is worse when it really wasn’t the producing company’s fault.

Verizon has a NFL deal and if you were trying to watch NFL on their network from a different legit source they could slow you down or just not provide reliable service so you would get a bad feeling about what you’re paying for and dump it for verizon’s.

In the past things like this has happened, and it would make a triumphant return if their lobbyists get their way.

This should not be a divided issue. There is literally no benefit to the customers, only the freedom for the companies to do what they do best… Try to get all the money they can from us pathetic chumps. From what I have seen our current government is not remotely listening to the people at this time, they are setting up their friends and other wealthy donors with the bonuses that they really want. We have to do what we can, call your politicians… It’s pretty much all we can do.

https://www.battleforthenet.com/ Get more information here.

One of the game servers I set up a decade ago is back for the winter. Updated with TitanRPG for better leveling. Fraghouse Nations is a group of custom mods that weren’t necessarily designed to work together all into one unique experience. It uses standard invasion but with a lot more monsters and on custom maps designed for this mode. There are customized Vehicles, Weapons, Maps, Monsters, and more?! The leveling is set up so you can get to a point where it’s relatively easy within about 45 minutes.

You can find this server under the Assault Tab where you will find the other legacy PWC-Gaming servers.