MicroATX Home Server Build– Part 4

After a longish break, here’s the next installment! So the server has been in production now since last September, and is running very well. After the previous post, this is what’s happened:

  • Installed ESXi 6.0 update 1 + some post u1 patches
  • Installed three VMs: Openbsd 5.8 PF router/firewall machine, Windows Server 2016 Technical Preview to run Veeam 9 on and an Ubuntu PXE server to test out PXE deployment
  • Added a 4 port gigabit NIC that I got second hand

In this post, I’ll be writing mostly about ESXi 6.0 and how I’ve configured various things in there.

For the hypervisor, I bought a super small USB memory, specifically a Verbatim Store n’ Stay (I believe this is the model name) 8GB, which looks like a small Bluetooth dongle. It’s about as small as they get. Here’s a picture of it plugged in:

The Verbatim Store N Go plugged in
The Verbatim Store N Go plugged in

Using another USB stick created with Rufus, which had the ESXi 6u1 installation media on it, I installed ESXi on the Verbatim. Nothing worth mentioning here. Post-installation, I turned on ESXi Shell and SSH, because I like having that local console and SSH access for multiple reasons, one of them I’ll get to shortly (hint: it’s about updating).

Since I didn’t want to use the Realtek NIC on the motherboard to do anything, I used one of the ports on the 4 port card for the VMkernel management port. One of the ports I configured as internal and one as external. The external port is hooked up straight to my cable modem, and it will be passed through straight to the OpenBSD virtual machine, so it can get an address from the service provider. The cable modem is configured as a bridge.

The basic network connections therefore look like this:

Simple graph of my home network
Simple graph of my home network

After the installation, multiple ESXi patches have been released. Those can be found under my.vmware.com, using this link: https://my.vmware.com/group/vmware/patch#search. Patches for ESXi can be installed in two ways: either through vCenter  Update Manager (VUM) or by hand over ssh/local esxi shell. Since I will not be running vCenter Server, VUM is out of the question. Installing patches manually requires you to have a datastore on the ESXi server where you can store the patch while you are installing. The files are .zip files (you don’t decompress them before installation), and are usually a few hundred megabytes in size.

To install a patch, I uploaded the zip file to my datastore (in this case the 2TB internal SATA drive) and through SSH logged on to the host. From there, you just run: esxcli software vib install -d /vmfs/volumes/volumename/patchname.zip

Patches most often require reboots so prepare for one, but you don’t have to do it right away.

Update 2 installed on a standalone ESXi host through SSH
Update 2 installed on a standalone ESXi host through SSH

Edit: As I’m writing this, I noticed Update 2 has been released. I’ll have to install that shortly..  Here’s the KB for Update 2 http://kb.vmware.com/kb/2142184

A one-host environment is hardly a configuration challenge, but some of the stuff that I’ve set up includes:

  • Don’t display a warning about SSH being on (this is under Configuration -> Advanced Settings -> UserVars -> UserVars.SuppressShellWarning “1”)
  • Set hostnames, DNS, etc. under Configuration -> DNS and Routing (also made sure that the ESXi host has a proper dns A record and PTR, too; things just work better this way)
  • Set NTP server to something proper under Configuration -> Time Configuration

For the network, nothing complicated was done as mentioned earlier. The management interface is on vmnic0, vswitch 0. It has a vmkernel port which has the management ip address. You can easily share management and virtual machine networking if you want to, though that’s not a best practice. In that scenario, you would create a port group under the same vswitch, and call it something like Virtual Machine port group for instance. That port group doesn’t get an IP, it’s just a network location you can refer to when you are assigning networking for your VMs. What ever settings are on the physical port / vswitch / port group apply to VMs that have been assigned to that port group.

By the way, after the install of Update 2, I noticed something cool on the ESXi host web page:

Host..client?
VMware Host..client?

Hold on, this looks very familiar to the vSphere web client which has been available for vCenter since 5.1?

Very familiar!
Very familiar!

Very familiar in fact! This looks awesome! Looks like yet another piece that VMware needs to kill of the vSphere Client. Not sure I’m ready to give it up just yet, but the lack of a tool to configure a stand-alone host was one of the key pieces missing so far.

Host web client after login
Host web client after login

In the next  post I will be looking at my VMs and how I use them in my environment.

Relevant links:

https://rufus.akeo.ie/
http://www.verbatim.com/prod/flash-memory/usb-drives/everyday-usb-drives/netbook-usb-drive-sku-97463/
The Host UI web client was previously a Fling, something you could install but that wasn’t released with ESXi https://labs.vmware.com/flings/esxi-embedded-host-client
But now it’s official: http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-esxi-60u2-release-notes.html

MicroATX Home Server Build – Part 3

Because I am impatient, I went ahead and got a motherboard, processor and memory. The components that I purchased were:

  • Asrock H61M-DGS R2.0 (Model: H61M R2.0/M/ASRK, Part No: 90-MXGSQ0-A0UAYZ)
  • 16 GB (2x8GB) Kingston HyperX Fury memory (DDR3, 1600MHz, HX316C10FBK2/16, individual memories are detected as: KHX1600C10D3/8G)
  • Intel i3-2100 (2 cores, with hyperthreading)

I ended up with this solution because I realized I may not have enough money to upgrade my main workstation, to get the parts from that machine into this one. I also didn’t have the funds to get a server grade processor, and getting an mATX server motherboard turned out to be difficult on short notice (did I mention I’m an impatient bastard?).

I ended up paying 48€ for the motherboard, 45€ for the processor (used, including Intel stock cooler) and 102 bucks for the 16GB memory kit.

The motherboard has the following specs:

  • 2 x DDR3 1600 MHz slots
  • 1 x PCIe 3.0 x16 slot
  • 1 x PCIe 2.0 x1 slot
  • 4 x SATA2
  • 8 USB 2.0 (4 rear, 4 front)
  • VGA and DVI outputs

The factors that led to me choosing this motherboard were mainly: Price, availability, support for 2nd and 3rd generation Intel Core processors (allowing me to use the i3 temporarily, and upgrade to the i5 later if I feel the need), and the availability of two PCIe slots. All other features were secondary or not of importance.

The reductions in spec that I had to accept were: No support for 32GB memory (as mentioned in the previous post), no integrated Intel NIC (this has crappy Realtek NIC, but I might still use that for something inconsequential as management; probably not though)

These pitfalls may or may not be corrected a later date when I have more money to put toward the build, and patience to wait for parts.

The CPU is, as mentioned, an Intel i3-2100. It’s running at 3.1 GHz, has two cores, four threads (due to HT), 3MB Intel ‘SmartCache’, and a 65W TDP.  It does support 32GB of memory on a suitable motherboard. I doubt the CPU will become a bottleneck anytime soon, even though it is low-spec (it originally retailed for ~120€ back when it was released in 2011). The applications and testing I intend to do is not CPU heavy work, and since I have four logical processors to work with in ESXi, I can spread the load out some.

Putting it all together

Adding the motherboard was fairly easy. There were some standoffs already in the case, but I had to add a few to accommodate the mATX motherboard. Plenty of space for cabling from the PSU, and I paid literally zero attention to cable management at this point. The motherboard only had two fan headers: One for the CPU fan (obviously mandatory..) and one for a case fan. I opted to hook up the rear fan (included with the case) to blow out hot air from around the CPU. I left the bottom fan in, I may hook it up later, or replace it with the 230mm fan from Bitfenix.

Initially, I did not add any hard drives. ESXi would run off a USB 2.0 memory stick (Kingston Data Traveler 4GB), and the VMs would probably run from a NAS. I ended up changing my mind (more on this in the next post). For now, I wanted to validate the components. I opted to run trusty old MemTest86+ for a day or so. Here’s the build running MemTest:

Build almost complete, running MemTest86+
Build almost complete, running MemTest86+

Looks to be working fine!

Here’s a crappy picture of the insides of the case, only covered by the HDD mounting plate:

Side panel open, showing HDD mounting plate, side of PSU
Side panel open, showing HDD mounting plate, side of PSU

One thing to note here is that if you want the side panel completely off, you need to disconnect the cables seen to the front left. These are for the power and reset buttons, USB 2.0 front ports and HDD led. They are easy to remove, so no biggie here.

One note on the motherboard: There has only ever been one release of the BIOS, version 1.10. This was installed at the factory (obviously, as there were no other versions released at the time of writing). If you do get this board, make sure you are running the latest BIOS. Check for new versions here: http://www.asrock.com/mb/Intel/H61M-DGS%20R2.0/?cat=Download&os=BIOS

So this is the current state of the build. Next up…

  • Installing ESXi 6.0U1 (just released in time for this build)
  • Deciding on where the VMs would run
  • Adding NIC and possible internal storage
  • Configuring ESXi
  • Installing guest VMs

Stay tuned!

Relevant links:

http://ark.intel.com/products/53422
http://www.asrock.com/mb/Intel/H61M-DGS%20R2.0/

http://www.kingston.com/datasheets/HX316C10FBK2_16.pdf
https://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-esxi-60u1-release-notes.html

MicroATX Home Server Build – Part 2

First an editorial correction to the previous post. An Intel B85 chipset motherboard will not support my current LGA1155 socket i5 processor, because that chipset is meant for the 4th Generation stuff (i.e. Haswell). Forget I wrote that.

And meanwhile, back at the content:

The case arrived last friday, and it’s a nice one! I’ve already stripped out the stuff I don’t need (mainly the 5,25″ bay internals), and installed the Corsair VX 450W PSU I had laying around from a previous build. A few notes:

  • The PSU installation was tricky
  • There are plenty of fans included, but they can easily be replaced. I’m thinking of getting their own 230mm (!) fan for the bottom of the case, since it should be fairly quiet
  • The handles on the bottom and top are a mixed bag. They are flexible, yet solid, so I wouldn’t worry about breaking them per se. I did end up removing the bottom handles (are they still handles even though they are in the bottom?) because it felt wobbly with them. I don’t want it to sway if i touch it.
  • Plenty of slots for 2.5″ and 3.5″ HDD’s. Very nice! All with removable mounting brackets of sorts
  • The case was wider than i thought, but this isn’t a bad thing
  • Most things are toolless, but there were some (easily removable) screws for certain parts
  • A nice selection of screws, rubber grommets, standoffs and other bits and bobs were included

The PSU installation

The PSU is installed in the front of the case, but not like you would think. I am not entirely sure why they opted for this method, but there is a standard power cable running from the from the front, from under the case, to the rear of the case, to a standard power plug. This is so that you can have all cables running to the rear of the case, even though the PSU isn’t physically in the rear.

The problem is that when you mount the PSU (it’s mounted top down, instead of on its side like usual), the regular power plug which you would plug to a wall outlet, is for the internal run, that ends with a 90 degree angled plug. It was *very* hard to fit, as you can see from the pictures. If your PSU has the plug near the edge of the PSU, it might even be impossible to fit the cable.

Detail of case bottom. Note PSU placement and power cable.
Detail of case bottom. Note PSU placement and power cable. Also note the rubber feet to lift the case slightly and allow at least minimal airflow below the case

From a space utilization perspective, I see why they did this. But the practicalities are well.. not. I seriously hope I didn’t break the cable, because the fit is so tight. If i did, it’ll probably blow a fuse the minute I turn it on, since it’s it would then be in contact with the metal of the case, causing a short.

Removing the bottom handle

I’m not sure why they made the bottom handle rounded too. The top one I get. It’s ergonomic, it looks good. But the bottom? I don’t want the case to be a rocking chair. I want it to sit still on the floor, shut up, and to what it is told.

Luckily, removing the bottom handles is an easy task: Remove four screws, pull it slightly and lift it out. The result isn’t pretty, but then, this is a home server build, not a beauty pageant. Someone asked Bitfenix if they’d consider different kinds of handles, or kits to cover the void left by removing a handle. The answer at the moment seems to be no, and I understand. As they say in the post, plastic is cheap, but making new molds isn’t.

I suppose if the visuals are a dealbreaker for you, either leave the handles in, or cover it with black gaffer tape or something. You can see the end result in the pictures.

Because of airflow, and stability, I added rubber feet to the bottom of the case. They seem to work fine. Whether I need more of a gap between floor and case remains to be seen. I have bigger rubber feet, and I’ll replace them if it seems necessary.

Lower case after handle is removed.
Lower case after handle is removed.
Full side view of case after lower handle removed
Full side view of case after lower handle removed

..and for my next trick

I am currently looking for a motherboard. I’m basically down to two choices:

Intel DH61ZE – https://www-ssl.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-dh61ze.html

A cheaper desktop board with the same H61 chipset: http://www.asrock.com/mb/Intel/H61M-DGS%20R2.0/ or https://www.asus.com/Motherboards/H61MK/specifications/

Price for the former is ~80€, price for the latter: 45-60€

I might not move the i5-2500 to this board after all. I’ve been looking at a used i3-2100, which has 2 cores and hyperthreading, making it nice for an ESXi box. They are priced at around 40-60 euros used.

Memory will come from an existing stash, but will be limited to 16GB due to the motherboards. Just something I’ll have to live with unless i dish out more money for a modern board, or a proper server grade board and processor.

MicroATX Home Server Build – Part 1

Today I officially started my new home server build by ordering a case. The requirements for building a new home server are the following:

  • It needs to be physically small
  • It needs to be able to operate quietly
  • It needs to utilize some current hardware to reduce cost
  • It needs to be able to run VMware ESXi 6
  • Needs to support 32GB RAM for future requirements
  • Needs to accommodate or contain at least 2 Intel Gigabit NICs

Having run a number of machine at home in the past three decades, some of these have become more or less must-haves. Others are more of a nice-to-have. I’ve had some real server hardware running at home, but most of the hand-me-down stuff has been large, powerhungry and/or loud to the point where running it has been a less than pleasurable experience.

The last candidate was an HP Proliant 350 G5 (or so?), which was otherwise nice, but too loud.

You will note that power isn’t a requirement. I don’t care, really. My monthly power bills for a 2.5 person household of 100 m^2 is in the neighborhood of a few dozen euros. I really don’t know, or care. I’m finally at a position where I can pick one expense that I don’t have to look at so closely. For me, that expense is power. Case closed.

The conditions I’ve set forth rule out using a classic desktop machine cum server thing. Those are usually not quiet, they use weird form factors for the motherboard, seldom support large amounts of RAM etc. etc. A proper modern server can be very quiet, and quite scalable as most readers will know. A new 3rd or 4th generation Xeon machine in the 2U or Tower form factor can be nigh silent when running at lower loads, and support hundreds of gigabytes of RAM. They are, however, outside my price range, and do not observe the “Needs to utilize some current hardware to reduce cost”-condition.

Astute readers will also pipe up with, “Hey, this will probably mean you won’t use ECC memory! That’s bad!”. And I’ll agree! However, ECC is not a top priority for me, as I am not running data or time sensitive applications on this machine. Data will reside elsewhere, and be backuped to yet another “elsewhere”, so even if there is a crash, with loss of data (which is still unlikely, even *with* non-ECC memory), I’ll just roll back a day or so, not losing much of anything. A motherboard supporting ECC would be nice, but definitely not a requirement.

Ruling out classic desktop workstations and expensive server builds I am left with two choices:

  1. Get a standard mATX case + motherboard
  2. Get a server grade mATX motherboard and some suitable case

The case would probably end up being around the same choice, as the only criteria is that it is small, and can accommodate fans that are quiet (meaning non-small fans). The motherboard presents a bigger question, and is one that I have yet to solve.

I could either go with a Supermicro, setting me back between 200-400 €, and get a nice server grade board, possibly with an integrated intel nic, out of band management etc., or I could go with a desktop motherboard that just happens to support 32GB of memory. There are such motherboards around for less than 100€ (For instance, Intel B85 chipset motherboards from many vendors).

Here’s the tricky part: I could utilize my current i5-2500 (Socket LGA1155) in this build, and associated memory. This would mean that the motherboard would obviously need to support that socket. Note! The 1155 socket is not the current Intel socket. We’re now at generation 6 (Skylake), which uses an altogether different socket (Socket 1151), which is not compatible with generations 2&3 (which used 1155), generation 4&5 (which used 1150).

Using my current processor would save some money. Granted, I’d have to upgrade the machine currently running that processor (meaning a motherboard, cpu and memory upgrade, probably to Haswell or Broadwell, i.e. Socket 1150), meaning the cost would be transferred there. But then again, I tend to run the most modern hardware on my main workstation, as it’s the one I use as my daily driver. The server has usually been re-purposed older hardware.

Case selection

I’ve basically decided on the form factor, which will be micro ATX (or mATX or µATX or whatever), so I can go ahead an buy a case. Out of some options, I picked something that is fairly spacey inside, and somewhat pretty on the outside, which doesn’t cost over 100€. The choice I ended up with was the Bitfenix Prodigy mATX Black.

Here’s the case, picture from Bitfenix (all rights belong to them etc.):

bitfenix_prodigy

Some features include:

  • mATX or mITX form factor
  • 2 internal 3.5″ slots
  • Suitable for a standard PS2 standard ATX PSU (which I happen to have lying around)
  • Not garish or ugly by my standards

I ordered the case today from CDON, who had it for 78,95€ + shipping (which was 4,90€). Delivery will happen in the next few days.

The current working idea is to get an mATX motherboard which supports my i5-2500 and 32GB of DDR3 memory. I’ve been looking at some boards from Gigabyte, Asrock and MSI. MSI is pretty much out, just because I’ve had a lot of bad experience with their kit in the past. May be totally unjustified, but that’s the way it feels right now.

I haven’t still ruled out getting a Supermicro board, something like this one: http://www.supermicro.nl/products/motherboard/Xeon/C202_C204/X9SCM-F.cfm but that would rule out using my current CPU and memory. I’d have to get a new CPU, which, looking at the spec, would either be a Xeon E3 or a 2nd or 3rd generation i3 (as i5’s and i7’s are for some reason not supported). i3 would probably do well, but I would take a substantial CPU performance hit going from Xeon or i5 down to i3. I’d lose 2 cores at least, which are nice to have in a virtualized environment, such as this.

Getting the board would set me back about 250€ and the CPU, even if I got it used would probably be around 100€. Compare this against an 80-100€ desktop motherboard, use existing CPU, existing memory (maybe?). Then again, I’ll have to upgrade my main workstation if I steal the CPU from there. Oh well. More thinking is in order, me thinks.

 

Last minute edit:

The hardware I have at my disposal is as follows:

  • Intel NICs in the PCI form factor
  • Some quad-NIC thing, non intel, PCIe
  • Corsair ATX power supply
  • Various fans
  • If I cannibalize my main rig:
    • i5-2500
    • 16GB DDR3 memory (4x4GB)