Patching ScaleIO 2.0 VMware Hosts

Recently did an ESXi patch run, along with some BIOS and firmware updates on a ScaleIO 2.x environment (more precisely 2.0.5014.0). The environment consists of some Dell PowerEdge servers, some of which are ESXi 6.0 build 3380124, some are Linux based, non-virtualized hosts. Luckily this environment was ScaleIO 2.x, because this version has a real maintenance mode in it (1.3.x did not). This means that while I can only patch one host at a time in this layout, I can do it fairly quickly and in a controlled fashion.

ScaleIO Maintenance Mode vs. ESXi Maintenance Mode

These are, obviously, two different things. With ScaleIO maintenance mode, you can put one SDS (providing storage services) host (at least in this configuration with two MDM’s) at a time into maintenance mode, which does not have an adverse impact on the cluster. The remaining SDS will take care of operations, provided it too does not break or go down at the same time.  After you are done patching, you exit maintenance mode, which the makes sure all changes are rebuilt and synced across the cluster nodes. This takes some time depending on the amount of data involved.

ESXi maintenance mode on the other hand, deals with putting the VMware hypervisor layer into maintenance mode so you can patch and perform other operations on it with no VMs running. The order is:

  1. ScaleIO
  2. VMware ESXi

And when coming out of the maintenance break, it’s the reverse.

I left the SVM (virtual machine on the host which takes care of the different functions that the host has, technically a SLES appliance) that I was patching, but I powered it down gracefully before putting the host into maintenance mode.

So accounting for all these things, my order was:

  1. Migrate all running VMs except the SVM off of the host using vMotion
  2. When the host is empty (bar the SVM), put ScaleIO into maintenance mode
    1. This is done via the ScaleIO GUI application, on the Backend page, by right clicking on the host. I did not have to use the force option, and neither should you…
  3. Shut down the SVM via “Shut Down Guest” in vCenter
  4. Put the host into maintenance mode without moving the SVM off the host (I suppose you could move it, but I didn’t)
  5. Scan and Remediate and install other patches (I installed BIOS, iDRAC and some other various updates via iDRAC; I had set them to “Install next reboot” so they would be installed during the same reboot as ESXi does remediation)
  6. Once you are satisfied, take the host out of maintenance mode
  7. Start the SVM on that host
  8. Wait for it to boot
  9. Exit ScaleIO maintenance mode (see 2.)
  10. Check to see that rebuild goes through (ScaleIO GUI application, either the Dashboard or Backend page)
  11. Make sure all warnings and errors clear. During host remediation and patching, I had the following errors
    1. High – MDM isn’t clustered (this is because you’ve shut down one of the SVMs containing the MDM role)
    2. Medium – SDS is disconnected (for the host being remediated)
    3. Low – SDS is in maintenance mode (for the host being remediated)
  12. After the SVM starts, it should clear all but the last alert, and once you have Exited Maintenance Mode, the final alert should clear
Exiting maintenance mode in ScaleIO GUI application
Rebuilding after exiting maintenance mode in ScaleIO

(Expected) Alerts during maintenance

As mentioned, you will have alerts and warnings during this operation. I had the following:

  • First, when putting the SDS into maintenance mode in ScaleIO, one warning about SDS being in maintenance mode:
SDS still on, ESXi not in maintenance
  • After SVM is shut down and ESXi is also placed in maintenance, two more:
All three alerts after host is in maintenance and SVM has been shut down
  • Then once you have remediated and taken the host out of maintenance, and started the SVM, you’re back to one, as in the first picture.
  • When you take the SDS out of maintenance, it will clear the last alert

Note that the highest rated alert, the Critical “MDM isn’t clustered” is actually noteworthy. It means that the SDS you are taking down for maintenance had the MDM role (critical for management of ScaleIO). Normally you’d have another one, and you shouldn’t proceed with any of this if you can only find one MDM, or if you already had this (or any other alert).

EMC has this to say about MDM’s (also see the document h14036-emc-scaleio-operation-ensuring-non-disruptive-operation-upgrade.pdf):

Currently, an MDM can manage up to 1024 servers. When several MDMs are present, an SDC may be managed by several MDMs, whereas, an SDS can only belong to one MDM. ScaleIO version 2.0 and later supports five MDMs (with a minimum of three) where we define a Master, Slave and Tie-breaker MDM.

Roles / Elements in ScaleIO

You can see the installed roles in VMware in the notes field, like so:

Roles in the Notes field in VMware

Elements or roles are (may not be a complete list):

  • MASTER_MDM – Master MDM node, Meta Data Manager, enables monitoring and configuration changes
  • SLAVE_MDM – Secondary MDM node, will take over if Master is unavailable
  • SDS – Storage node, ScaleIO Data Server, provides storage services through HDD, SSD, NVMe etc.
  • SDC – ScaleIO Data Client, consumer of resources (e.g. a virtualization host)
  • RFCACHE – Read-only cache consisting of SSD or Flash
  • RMCACHE – RAM based cache
  • LIA – Light installation agent (on all nodes, creates a trust between node and Installation Manager)
  • TB – Tiebreaker, in case of conflicts inside cluster, counted as a type of MDM, non critical except in HA/conflict situations

ESXi funny business…

While running remediate on the hosts, every single one failed when installing patches.

Scary Fatal Error 15 during remediation

A very scary looking Fatal Error 15. However, there’s a KB on this here.

So, (warm) reboot the host again, wait for ESXi to load the old pre-update version, and do a re-remediate without using the Stage option first. I used stage, as I’m used to, apparently this breaks. Sometimes.

And to re-iterate, I was patching using vCenter Update Manager (or VUM) from 6.0 build 3380124 to 5050593.

Sources

docu82353_ScaleIO-Software-2.0.1.x-Documentation-set.zip from support.emc.com (not actually for the version in use, but similar enough in this case. Use at your own risk..

ScaleIO v2.0.x User Guide.pdf contained in the above mentioned

https://community.emc.com/thread/234110?start=0&tstart=0

https://www.emc.com/collateral/white-papers/h14344-emc-scaleio-basic-architecture.pdf

https://www.emc.com/collateral/white-papers/h14036-emc-scaleio-operation-ensuring-non-disruptive-operation-upgrade.pdf

Home Lab Xeon

The current home lab setup consists of an Intel Core i3-2100 with 16GB of DDR3, a USB drive for ESXi (on 6.5 right now) and a 3TB WD for the VMs. While the Intel i3 performs perfectly for my needs, I came across a Xeon E3-1220 (SR00F, Ivy Bridge), which should be even better!

For the specs, we have the following differences:

Model Intel Xeon E3-1220 Intel Core i3-2100
Released: Q2-2011 Q1-2011
Manufacturing process: 32nm 32nm
Price originally: 189-203 US dollars (more in euroland) 120 USD
Core count: 4 Cores 2 cores
Hyperthreading No Yes
Base Freq: 3.10 GHz 3.1 GHz
Turbo Freq: 3.40 GHz No
TDP: 80 W 65W
Max Memory: 32 GB ECC DDR3 32 GB Non-ECC DDR3
L1 Cache: 128 + 128 KB 64 + 64 KB
L2 Cache: 1 MB 512 KB
L3 Cache: 8 MB 3 MB

So we can see that the Xeon part is 4 core processor, without hyperthreading, so real cores as opposed to the i3’s threads. It’s more power hungry, which is to be expected, but can also Turbo at a higher frequency than the i3. Also, the Xeon has more cache, which is also to be expected with a server grade component.

A notable thing is that the Xeon, being a server part, does not include the GPU components, so I’ll have to add a GPU at least for the installation. I run the server headless anyway, but I want to see it POST at least. I think I’ll have to add a PCI card for this it has no PCI slots so, as I only have one PCIe slot (well there are some x1 slots but I have no such cards), and that’s used by the NIC. The motherboard is an Asrock H61M-DGS R2.0 which has one x16 slot and one x1 slot. Maybe I’ll do it all headless and hope it posts? Or take out the NIC for the installation?

Some yahoo also tried running an x16 card in an x1 slot here. Might try that but since I have to melt off one end of the x1 slot, probably not.

There are apparently some x1 graphics cards, but I don’t have one as I mentioned. An option could be the Zotac GeForce GT 710, which can be had for 60 euros as of this post.

Preparations

I went to the pharmacy to get some pure isopropyl alcohol. It wasn’t on the shelf, so I had to ask for it. I told the lady I need some isopropyl alcohol, as pure as possible. She looked at me funny and said they had some in stock. I told her I’m using it to clean electronics, so she wouldn’t suspect I’m some sort of cringey soon-to-be-blind  (not sure if you get blind from this stuff, but it can’t be good for you) wannabe alcoholic, to which she replied that she doesn’t know what i’ll do with it, or how it will work for that. She got the bottle, which is described as “100 ml Isopropyl Alcohol”. There is a mention of cleaning vinyl disks and tape recorder heads on the back, so I was vindicated. There’s no indication of purity on the bottle, but the manufacturer lists above 99.8% purity here. Doesn’t exactly match the bottle, but it’s close.

Why did I get isopropyl alcohol? Well, because people on the internet said it’s good for cleaning off residual thermal paste from processors and CPU coolers. With common sense 2.0, I can also deduce that anything with a high alcoholic content will evaporate, and not leave behind anything conductive to mess things up. Oh and it cost 6,30€ at the local pharmacy. It’s not listed on the website (or it says it’s no longer a part of their selection).

Let’s see how it performs. I’m using cotton swabs, but I suppose I could use a paper towel. If it leaves behind cotton pieces, I’ll switch to something else.

The Xeon originally had a passive CPU block and a bunch of loud, small case fans, but I will use the same cooler as for the i3.

Take out the i3 and the cooler. Clean the cooler off with the isopropyl:

Isopropyl worked wonders

Put in the E3, new thermal paste. I used some trusty Arctic Silver 5.

Termal paste added, note artistic pattern

Re-attach the cooler and we’re off to the races. I’ll note here that I hate the push through and turn type attachments of the stock Intel cooler. Oh well, it’ll work.

 

Powering on

Powering the thing on was the exciting part. Will there be blue smoke? Will it boot headless? Will it get stuck in some POST screen and require me to press a button to move on? Maybe even go into the BIOS to save settings for the new CPU?

Strangely enough, after a while, I started getting ping replies from ESXi meaning the box had booted.

There’s really nothing left to do. ESXi 6.5 recognizes the new CPU and VMs started booting shortly after.

Xeon E3 running on ESXi 6.5

Lessons learned – P2V Exchange 2007

I did a physical to virtual conversion of some Exchange 2007 servers, running on Server 2008 last weekend. While everything went fine in general, there were a few lessons to be learned. There’s a lot of forum threads and blog posts written about this topic, but I figured I’d put up some of my experiences anyway.

I will start by describing the environment. The old servers were running on physical Dell hardware, at a remote location. The connection to the new site was 1Gbps end-to-end. The new environment is a fresh vSphere 5.5 cluster. I used VMware Converter Standalone version 5.5.2 for the conversion. Due to the nature of the tool, and the source being a physical server, the conversion was done “hot”, with the source servers on.

The prep

I started out by doing an inventory of the source servers. Checked disk sizes, memory usage, cpu usage. Made note of each service running, and whether they were automatically started or not. One of the first things you notice after a conversion is that your event log isn’t a pretty sight. Certain hardware is always left over after a P2V, which will have to be removed. So I also made a note of any “special” hardware that might be running, that has to be removed after conversion. Things like usb devices, display adapters, disk controllers (SCSI), HBA’s, network cards etc.

Anything as intensive as Exchange (such as SQL, sharepoint, active directory), needs to shut up before doing something like a P2V. Otherwise you will end up with either a non-functional virtual machine, or inconsistencies or the like.

I started by unmounting the Mail DB’s, and DB’s for public folders. Just to be on the safe side. They will be unmounted when you shut down the services anyway, but I guess I’m just pedantic that way. Some guides suggest that you could just unmount the databases and then start the conversion. I wanted to be safe so…

The services I stopped on a 2007 machine with the CAS, Hub, and Mailbox roles were:

– Microsoft exchange active directory topology service
– Microsoft exchange transport log search
– Microsoft exchange transport
– Microsoft exchange service host
– Microsoft exchange search indexer
– Microsoft exchange replication service
– Microsoft exchange mail submission
– Microsoft exchange mailbox assistants
– Microsoft exchange file distribution
– Microsoft exchange anti-spam update
– Microsoft exchange information store
– Microsoft exchange system attendant
– Microsoft search (exchange)
– IIS admin service
– World wide web publishing service

I also stopped services for Backup Exec, and for the AV-product. I’ve noticed AV-products tend to mess with VMWare Converter, at least in some cases.

On the Edge-server I stopped the following services (in addition to BE and the AV-stuff):

– Microsoft exchange ADAM
– Microsoft exchange transport log search
– Microsoft exchange transport
– Microsoft exchange anti-spam update
– Microsoft exchange credential service

The conversion

Conversion ran on the machines itself, using the “Powered-on machine”-option, and selecting “This local machine”. Pretty much default settings. Finalize synchronization after conversion. Converted the hard drives to thin. No changes to running services or anything like that. I usually don’t install VMWare Tools automatically, and I don’t uninstall VMware Converter components automatically either. I don’t trust automatics, and I usually take care of those post-conversion by hand.

Conversion ran at a comfortable 20-40MB/s and was done in a reasonable time. Considering it’s VMWare Converter.

Post-Conversion

Every P2V conversion guide says: After conversion, shut down the old physical machine and disconnect it from the network to make sure it never comes online again. There is a reason for this. Due to the environment, and lack of OOB management (no iDRAC, ILO or the like) there was no way to shut down or remove it from the network completely, without losing the ability to rollback. You always kind of want the option to go back to the old server, in case your conversion really goes tits up.

Anyway, the original machine was renamed, dropped from the domain and dropped from all networks except one. And in that network, I changed the IP. This way I still had a way in if I needed, but nothing to point back to the old server. Right? Wrong.

Here’s where service principal names come in. SPN’s can mess things up very quickly unless you are careful. In this case, even though the old server was renamed, and removed / changed in all networks, there were still things referring to the old server, namely SPNs. There are a number of uses for them, for instance Kerberos authentication. An exchange server has a number of SPN records, not just the regular HOST/server.name ones. There were also records like SMTP/ and MAIL/ and EXCHANGE/. Even though I had rebooted the server, the old SPNs had not disappeared. New ones were simply added. I didn’t want to start Exchange to see if the records would be removed/changed at that point, so I simply deleted all the SPNs that still referenced the old server name. I left the ones pointing to the new name, as they would not conflict with anything.

I had records pointing to the old server name for all of the following records: One pointing to the current name (call it server_old) and the other pointing to server (the original pre-virtualization name):

spnt_vanhalla_maililla_CLEAN_2
Actual server names removed to protect the innocent

 

Prior to removing the records, the new converted virtual server would not log into the domain. The error I received upon login was: “Error: The security database on the server does not have a computer account for this workstation trust relationship.”

I was able to login using the local account, so I knew I wasn’t completely hosed. The error message led me on a wild goose chase, though. The server had a computer account under the correct name in the domain (on all domain controllers). I tried resetting the computer account, I tried removing it, dropping the server out of the domain and then back. No help.

Eventually I started looking at SPN records using ADSIedit on one of the DCs. Under the domain context, find the computer account for the old account, and look under serviceprincipalname. Remove the SPNs from the old physical server compunter account that are pointing to the new virtualized server. Reboot the new virtual machine. There should be no conflicting names anywhere in the domain, and the login should now work. As it did!

The Trouble with Tumblr…and other stuff

So.. I’ve been thinking about two things in relation to Tumblr, the popular image-blogging site.

1) Why is it so hard to get an image in the original size? Sure, this may be theme related stuff, I’m not tumblr’d enough to say. But when you see a thumbnail on a site, shouldn’t you just be able to click it, and get the original, right? This has been the kind of.. use case since the very early days of the internet. The point of the thumbnail is to, one, decrease load times by displaying a smaller ‘preview’ image first, and letting the user decide whether he wants to load the larger image, and two, to save layout space on your site by not covering the entire screen with one image. The case in Tumblr is often that I’m clicking on an image, and then I’m taken to the comments page, where I can either click the “source” link under the image, or the link for the person (‘via xxxxx’) who reblogged the image from the original poster. Then I might get the large, original picture. Or not! I find this extremely disturbing. If  it is a theme issue, then okay, fine. But then most people are using very broken themes. It also might signify that most people have no idea how to fix the theme, or even what makes up a ‘theme’ on Tumblr. Which might, or might not, say something about the blogger on Tumblr. But enough about this angle! I digress!

2) Who provides the original content? Pick any tumblr, save for say, the official Tumblr page for a celebrity or so. Look at the images. Are they all reposts/reblogs of some other image? In some cases the reblog chain for an image is stuponfuciously long. Is there original content on Tumblr, or is like, everything a reblog of a reblog of a reblog of a reblog of some picture someone found somewhere, which was still not the original source?

Okay, I realize this is a silly thing to get annoyed over, but that’s me.. for you.

On to other things!

I’m moving. Again. I seem to live in one apartment for two to three years. But this time, it’ll be different! It’s a sweet pad. Built 2011. Four rooms, a big washroom and sauna. 98 m^2. Huge living room (I’m looking into the crystal ball and I’m seeing, yes.. a projector…). And, again, a hacking room. Same as in my last apartment. I missed that place. A room that I can fill top-to-bottom with hardware, books, whatever. A place where I can sit down, close the door and do whatever. I’m getting fuzzies just thinking about it. It’ll be great. Also, nobody will be disturbed by the humming. It’ll just be there, and it’ll be sweet.

What else what else. Didak has posted some new pics of his famous Home Office, version 7. They are the sweetness. Check them out. Waiting for a writeup or something, or a making of article. I’ve really enjoyed those in the past.

I wanted ESXi 5.5 on some Dell and HP boxes. I had no joy booting from a USB that was made using Unetbootin or Win32DiskImager. It simply wouldn’t boot. Now note, that the same image would eventually boot correctly via ILO/iDRAC using the virtual media feature. It might be a problem with the USB media I was using. Or the software that I’m using to create the bootable media. Or the specific server hardware, or the BIOS/UEFI settings of them, or UEFI in general. I googled for a solution, and I found one. Here it is! Following those instructions, I now have a proper bootable (on any machine I’ve encountered so far) media, with ESXi 5.5 on it. It might be helpful for you. Also, remember to use the vendor specific media for both HP and Dell, and not the generic VMWare Image. They contain  diagnostic tools, drivers and other stuff that will be useful later. You can find the vendor specific bootable media for HP and Dell in those two links there. These may or not be current, but they’ll take you somewhere. For Dell, google for the esxi version, and then A0x, where x is a number. When I was installing, the latest was A01.

What I’ve been reading lately: Tom Clancy’s Threat Vector (his last book?). Okay for a Clancy, and pretty eerily realistic. After that I started on Neal Stephenson’s The Diamond Age. Which has been moving a bit slowly at times. It goes from okay to excellent between chapters, so sometimes I’m reading twenty pages in one go, and sometimes it’s more like sixty or eighty. It’s a curious book, that. There are absolutely brilliant parts, and then some parts that are, to put bluntly, boring. But I’ve been meaning to read that for a while now, and I’ll be happy to finish it soon. Snow Crash was excellent, and so was Reamde. After this I will either read The Baghdad Blog, by ‘Salam Pax’, or another Clancy perhaps? I have like five books on my reading shelf.

 

Removing trickier VMFS-datastores

Ok, maybe tricky isn’t the right word, but at least I couldn’t find anything written on this particular issue. Maybe it’s too simple a solution even for the VMware KB, but anyway.

I was cleaning out some local datastores (Smart Array 420 and 420i controllers) and ran into an issue where I was unable to remove the VMFS datastore because of a file in use error. It didn’t give me specifics; just told me that there were file(s) in use, and/or that the datastore was busy. After a fair amount of googling I started throwing some commands at it through the ssh. There’s a vmkfstools command that can break any existing locks, and it warns you that it will do it forcibly. So I tried that, given that there was nothing on the datastore that I couldn’t afford to lose (the point, after all, was to remove it). Despite grave warnings, vmkfstools was unable to break the lock and didn’t really give me a proper reason.

Looking at the vmkernel logs (/var/log/vmkernel.log by default), I saw the same references to files being in use, but no exact reference as to what files and where. No virtual machines were running anymore, and I had deleted most everything that I could off the datastore by hand already. There was a rather specific error message relating to corruption, and googling that got me exactly diddley. The datastore had had some problems previously, some hardware had been replaced, so there were a lot of variables and things that could have affected the case.

The solution, how ever, was much simpler. ESXi (5.1 update 1), a standalone server not attached to any cluster, was shoving logfiles onto the datastore I was trying to remove. Obviously, there would be ‘file in use’-errors. D’uh. So, from the host level, I went to the Configuration tab, and from there Advanced Settings. From there, Syslog -> Syslog.global.logDir. If it is null (and it can be null), the logs are all reset if and when you reboot the host. If there’s a path, in the style of [datastore]/path, it’ll use that instead.

So for this particular case, I set a null path, which raises a warning that logs are being stored in a non-persistent location, but it then allowed me to delete the datastore (and/or detach it first) without issue.

I was probably thrown off by the vmkernel messages about corruption, though they may have played a part in why certain files and folders couldn’t be deleted by hand using datastore browser or the command line.

After everything was done, I redirected the logs back to one of the datastores, which clears the warning (no reboot needed here, or when I set the null path earlier).

I tried to find the specific error messages but I couldn’t. I may have them somewhere so I’ll shove them in here if I find them.

Some of the commands that helped me along were:

esxcli storage filesystem list ## This lists the filesystems that the server knows about, including their UUID, label and path. These are needed for many vmkfstools commands, so it’s a good place to start

vmkfstools -B /vmfs/devices/disks/naa.unique_disk_or_partition_goes_here ## This tries to ‘forcibly’ break any existing locks to the partition that may prevent you from proceeding. Didn’t work in my case, but also didn’t tell me anything useful..

vmkfstools -V ## re-read and reload vmfs metadata information

Some of the sites and blogs that helped me along:

VMWare KB article 1009570
VMWare KB article 2004201
VMWare KB article 2032823
VMWare KB article 1009565
http://blogs.vmware.com/vsphere/2012/05/vmfs-locking-uncovered.html
http://kb4you.wordpress.com/2012/04/23/unpresenting-a-lun-in-esxi-5/
VMWare KB article 2011220
VMWare KB article 2004605
http://arritdor.e-wilkin.com/2012/03/removing-vmfs-datastore.html

Thanks to everyone who wrote those.

Pi musings

So now I’ve gone and done it! I am doing something with my Pi. What I’ve done is, install nginx in a jail on it. Why? Just because I haven’t done that before. I’ll talk a bit more about what I did, and how in this post.

Why nginx? Well, the primary reason is that it’s growing in market share, and because I have very little hands-on experience of it. Also because I have this idea in my head that it’s slightly less bulky than say Apache2. Many Pi-specific pages also recommend lighthttpd, but since nginx is more prevalent on the net, I chose that.

Note! You could prepare the chroot environment beforehand. If you wish to do so, jump to the appropriate heading and then come back here. This is the order that I did things in, so if you, for some yahoo reason want to follow that, read on.

The Raspbian repositories contain a version of nginx, but it’s supposedly very old. I opted to compile from source, which seemed like a good idea after the repositories listed for a more current version didn’t work properly for the version of Raspbian / architechture of the Pi. Obviously, compiling on the Pi as a rather slow process, but this isn’t a rush order. To start off, i installed some necessary tools so I could compile from source:

sudo apt-get -y install wget build-essential libpcre3-dev libpcre++-dev zlib1g-dev libssl-dev

After this, wget the latest source package for ngingx, http://nginx.org/en/download.html, and unpack this to a location of your choosing:

wget http://nginx.org/download/nginx-1.5.6.tar.gz and the pgp signature: wget http://nginx.org/download/nginx-1.5.6.tar.gz.asc

Get the public key for the signer of the package (in t his case Maxim Dounin)  wget http://nginx.org/keys/mdounin.key

Import it: gpg –import mdounin.key

And finally run gpg nginx-1.5.6.tar.gz.acs

You should get a message about a good signature, however, it’ll not be a trusted signature. You can’t be sure it belongs to the owner. The key would need to be signed by trusted sources, in order to establish the web of trust properly. But for now, we are content.

Then once you are all wrapped in tin foil, go prepare a pot of your favorite coffee and start compiling nginx. Change, add, remove options as needed. This is just from another howto, so you might like different locations for your logs, or include modules that are not included here:

cd nginx-$VERSION ./configure –sbin-path=/usr/sbin/nginx \ –conf-path=/etc/nginx/nginx.conf \ –pid-path=/var/run/nginx.pid \ –error-log-path=/var/log/nginx/error.log \ –http-log-path=/var/log/nginx/access.log \ –with-http_ssl_module \ –without-http_proxy_module make

After this, you could potentially start nginx using /usr/sbin/nginx, but we’re not done yet.

Chroot

Here, we want to do some potential damage control. The webserver is living inside its own little world, and if someone gets into that world, it’s kind of small and boring, and has no real access to the underlying OS.

We can do this either manually, or by giving the chroot directory (the new root) as a variable:

D=/example
mkdir $D

After this, we need to create necessary directories inside the chroot directory for nginx to work properly.

# mkdir -p $D/etc
# mkdir -p $D/dev
# mkdir -p $D/var
# mkdir -p $D/usr
# mkdir -p $D/usr/local/nginx
# mkdir -p $D/tmp
# chmod 1777 $D/tmp
# mkdir -p $D/var/tmp
# chmod 1777 $D/var/tmp
# mkdir -p $D/lib

Note that we also give permissions to tmp and /var/tmp at this stage. Just to keep them writable by everyone just like they are in the base OS. Makes it easier for non-privileged users to write temporary files during installs or stuff needed when you are running the server.  Some instructions (like the one on Nixcraft that I relied on heavily while doing this) create a lib64 directory inside the chroot. I didn’t even have such a directory in the base Raspbian, so I followed suite inside the chroot by making a lib directory.

Next, create the following inside the chroot/dev directory, but first checking their special attributes using:

# ls -l /dev/{null,random,urandom}

You’ll get something like:

crw-rw-rw- 1 root root 1, 3 Jan  1  1970 /dev/null
crw-rw-rw- 1 root root 1, 8 Jan  1  1970 /dev/random
crw-rw-rw- 1 root root 1, 9 Jan  1  1970 /dev/urandom

Note column five. 1,3 and 1,8 and 1,9.  You need to set these attributes inside the chroot too. Do a:

# /bin/mknod -m 0666 $D/dev/null c 1 3
# /bin/mknod -m 0666 $D/dev/random c 1 8
# /bin/mknod -m 0444 $D/dev/urandom c 1 9

Next, you’ll copy all the nginx files from your base OS inside the chroot. For instance:

# /bin/cp -farv /usr/local/nginx/* $D/usr/local/nginx and

# /bin/cp – farv /etc/nginx/* $D/etc/nginx

Next a tricker part. Move all necessary libraries to run nginx to the chroot. You can find out what you need by doing a:

ldd /usr/sbin/nginx

You’ll get an output similar to:

/usr/lib/arm-linux-gnueabihf/libcofi_rpi.so (0xb6f94000)
libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6f6a000)
libcrypt.so.1 => /lib/arm-linux-gnueabihf/libcrypt.so.1 (0xb6f33000)
libpcre.so.3 => /lib/arm-linux-gnueabihf/libpcre.so.3 (0xb6ef2000)
libssl.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libssl.so.1.0.0 (0xb6ea2000)
libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0 (0xb6d3f000)
libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6d34000)
libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xb6d16000)
libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6cee000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6bbf000)
/lib/ld-linux-armhf.so.3 (0xb6fa1000)

All of these need to go to the corresponding locations inside the chroot. There are scripts floating around for checking what you need and copying them over; I just copied them manually because I’m a pleb.  You can always come back later; nginx and any other tools you use will tell you if you uare missing any libraries, and you can copy them later.

Copy the relevant contents of /etc to the chroot. I had problems with the users inside the chroot, but it might have been something I messed up. I was unable to run it using nobody:nogroup, and had to resort to using the uid and gid, but more on that later. If someone knows what I fucked up, and happens to read this, use the comments, thanks! But the copying I mentioned (again thanks to Nixcraft):

# cp -fv /etc/{group,prelink.cache,services,adjtime,shells,gshadow,shadow,hosts.deny,localtime,nsswitch.conf,nscd.conf,prelink.conf,protocols,hosts,passwd,ld.so.cache,ld.so.conf,resolv.conf,host.conf} $D/etc

And some directories (though my raspbian install didn’t have prelink.conf.d at all):

# cp -avr /etc/{ld.so.conf.d,prelink.conf.d} $D/etc

We’re just about done. Kill an existing nginx’s using pkill nginx or something like killall -9 nginx to do it more violently.  Then we can run a test of nginx inside the chroot. This will tell you what is missing (libraries, files etc.), or if your config syntax is wrong:

# /usr/sbin/chroot /nginx /usr/local/nginx/sbin/nginx -t

To run it finally, remove the -t at the end. As I mentioned, at this point I had issues about a line in the nginx config file (/etc/nginx/nginx.conf), which is “user nobody;”. For the life of me  I could not get it to run using this user, even though I had it inside the chroot/etc/passwd, and group files. It just told me unknown user and so on. Changing the user also had no effect, i tried creating a fresh user, but to no avail. Finally, I ended up running nginx with:

/usr/sbin/chroot –userspec=65534:65534 /nginx /usr/sbin/nginx

Where 65534 is the uid and gid (respectively) of nobody and nogroup. Note that we are chrooting into /nginx (my chroot directory for nginx) and then from there, running /usr/sbin/nginx which is the script that starts nginx. After this, we have nginx running under the correct user and group:

nobody    4355  0.0  0.1   4984   724 ?        Ss   Oct07   0:00 nginx: master process /usr/sbin/nginx
nobody    4356  0.0  0.2   5140  1228 ?        S    Oct07   0:00 nginx: worker process

To be absolutely sure that nobody runs the “base OS” version of nginx, you can remove the directories associated, or rename the executable file under /usr/sbin (i called mine nginx_nonchroot), so I can verify that file isn’t being run. Or remove the execute bit with chmod -x /usr/sbin/nginx.

When starting nginx at boot, be sure you are doing it in the right way to ensure it’s inside the chroot:

# echo '/usr/sbin/chroot /nginx /usr/sbin/nginx' >> /etc/rc.local

To verify that your nginx is running inside the chroot, use the process id (second column when you run ps aux | grep nginx; in my example, 4355), by running:

# ls -la /proc/4355/root/

…and you’re getting the contents of the chroot root, i.e. all the directories that sit under the chroot /

drwxr-xr-x 10 root root 4096 Oct  7 19:00 .
drwxr-xr-x 24 root root 4096 Oct  6 23:24 ..
drwxr-xr-x  2 root root 4096 Oct  7 19:11 bin
drwxr-xr-x  2 root root 4096 Oct  6 23:25 dev
drwxr-xr-x  5 root root 4096 Oct  7 19:43 etc
drwxr-xr-x  3 root root 4096 Oct  6 23:36 lib
drwxr-xr-x  2 root root 4096 Oct  7 00:03 run
drwxrwxrwt  2 root root 4096 Oct  6 23:23 tmp
drwxr-xr-x  5 root root 4096 Oct  6 23:27 usr
drwxr-xr-x  5 root root 4096 Oct  7 19:51 var

You can also change the default index page so you can see that that’s the one being loaded.  In my case /nginx/usr/local/nginx/html/index.html. You can reload the chrooted nginx using:

# /usr/sbin/chroot /nginx /nginx/usr/sbin/nginx -s reload

You could now make sure nginx is listening on your pi, by using:

netstat -pantu | grep nginx

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4355/nginx   

Browse to the ip assigned to your pi and see your webpage! Make sure you lock things down with iptables, and allow traffic only to ports that you want, and from addresses you want.

Infinite props to Nixcraft for this article, which helped me along the way. The main reason I wrote this was that my install  was slightly different, and I figure I’d type my own problems and solutions down. Also, raspbian has changed slightly (i guess?); So here you are. This howto was also very helpful, thanks to elinux.org.

 

 

 

LSI Updates and Pi

There’s no possible way to make a Raspberry Pi-joke that hasn’t already been made.

LSI

So far so good. Things’ve been working fine, though I have to look into disabling the bios since I’m not booting from any drives that are behind the LSI card. Boot times are three times as long as without the card, even though the OS is loading from the Samsung 840 Pro SSD drive.

I used MegaRaid Storage Manager for Windows to install the latest BIOS for my card. I went to the LSI site, searched for Host Bus Adapters -> LSI SAS 9211-8i -> Firmware, and downloaded the only available package (at the time this was named “9211-8i_Package_P17_IR_IT_Firmware_BIOS_for_MSDOS_Windows”, released Aug 09, 2013, the same package as for the IR-firmware installed in the previous post). Inside the archive, you will find various folders. Look in the  folder “sasbios_rel” and check that you have mptsas2.rom in there. That’s the BIOS image.

The good news is, as I mentioned, once you have the Storage Manager software installed, and your card is recognized, you can flash the BIOS from Windows without issues. This should also work for Firmware, but I haven’t tried this yet, as I am already running the latest IR-firmware. Open up SM, and somewhere in the middle you will find Update Firmware. There, select BIOS (middle selection for me), and browse to the folder mentioned earlier. Inside, select the mptsas2.rom file. Hit OK, and it will ask you to check a box and confirm that you want to update the BIOS. After that, it’ll flash, and tell you when it is done. It will show you the old BIOS version until you reboot. My card was 7.29.0.0, and is now 7.33.0.0. Improvements are minimal, but there were some.

One note on the Write Cache, mentioned in the last post. I was unable to enable this from Storage Manager. Perhaps due to the fact that there is no battery backup unit. I’ll have to look more into this at a later date.

PI

Got me a Pi. The B model, from local RS reseller, Yleiselektroniikka. Cost me 47 bucks including taxes. It’s the revised Model B, with 512MB memory. I also got a transparent case, which was 10 bucks. I didn’t get a powersupply, because I have plenty of USB chargers for various devices (and a few generic ones) that provide 1A+ @5V. My HTC Desire Z charger powered the Pi just fine, even though there’ve been reports of “flaky” mobile phone chargers not working with the Pi.

I have an 8 GB Verbatim SD-card for this project, and I dropped the latest NOOBS image from the Raspberry Pi homepage on the card, after formating the card FAT. I then installed Raspbian from the NOOBS-installer, and proceeded to do an apt-get update && apt-get upgrade, which also upgraded the Pi bootloader to the latest version (as was recommended by the small booklet that came with the Pi.)

I haven’t done much with the device yet (joining the club of Pi owners everywhere! :)), except hook things up and tried it out a bit. It works great! Or just as advertised. Obviously the boot is a little bit slow, but nothing out of the ordinary, considering the specs. HDMI out works fine; I use an HDMI -> DVI cable for this.

Blabbity blab

Nothing specific to talk about, but I felt like writing anyway.

Don’t multihome vmk ports in ESXi

Multihoming vmk ports on ESXi 5 (?) and later is not kosher. It’ll allow you to make the config, and it’ll even work, for a random period of time. You probably want separate physical ports for management and vMotion, so you’re bound to have two vmk ports, don’t put them on the same subnet/vlan. This was supported in ESX 4 and earlier, perhaps, but not in any later versions of the VMware hypervisor. This KB-article helped out a lot, as well as this quickhand on ESXi shell network commands. The setup was roughly the following:

  • vmk0 – management – vSwitch0 – 10.10.10.1
  • vmk1 – vmotion – vSwitch1 – 10.10.10.2

One host with this config dropped off the network, and the management port wouldn’t respond. The other vmk interface still responded perfectly, and the machines were on separate vmnics and vSwitches so they were unaffected as well. But vCenter lost connectivity to the host. Obviously, migrating the vm’s off the host was not an option, as there was no way to reach it through the vSphere client. The cluster did not have HA enabled.

To fix it, the steps were roughly:

  1. Enable ESXi Shell, if it isn’t already, through the DCUI -> Troubleshooting options -> Enable ESXi Shell
  2. Hit Alt-F1 to go to the shell
  3. Disable the vmnic that is not the management vmnic (in our example, vmk1, for vmotion) using esxcli network nic down -n vmnic   ##make sure you get the right vmnic, doublecheck in DCUI
  4. You can Alt-F2 back to DCUI and check out the network settings to verify that it’s down. Once the conflicting vmk is down, the primary one should start working, and you’ll have management back. If necessary, restart management agents / network from DCUI.
  5. There’s also esxcfg-vmknic -d (for delete, -D for disable) portgroup. To list the portgroups, use esxcfg-vmknic -l (and locate the conflicting, non-management vmk, and check the name of it)
  6. When management is restored (you can verify by running the Test Management Network in DCUI, and ping your management IP), do the rest from the vSphere Client (restoring what ever vmk you disabled, and the functionality it had (be it vmotion or so)). This time, make sure you use a separate subnet/vlan (not the same as for management)
  7. Also NOTE that if you used the ESXi Shell to disable a NIC, you have to enable it from there as well. I’ve found no way to say “vmnic up” in vSphere Client. If you know of a way please let me know in the comments. I had to make an extra trip to the data center to get the interface up, and then finalize the config in vSphere client.

Considering a Soekris or Mikrotik

For years (uh say, 8 years?) I’ve used an older workstation PC with two Intel 1Gbps NICs and lately, an SSD, plus OpenBSD & pf as my network firewall/router. It’s a rather clunky solution for a simple task, but it has served me well for years, without too many problems. After listening to TechSNAP (the latest couple of episodes, I guess), I’ve been thinking about replacing that box with a smaller solution, such as hardware from Soekris or Mikrotik. Soekris are a bit expensive, but they are perhaps.. more fully fledged than the Mikrotik. Both, as I understand, allow for your own choice of OS. I would still be running BSD (be it Free or Open), because that’s what I sort of trust with these matters. The other option is to buy an Atom board, slap on 2-4GB memory, two NICs (or a multiport NIC), and the SSD that I already have, and then run that in a smaller form factor case. I’m more of a do-it-yourself kind of guy, so I might end up going that route anyway.

Reading stuff

I’ve been reading a lot lately. Well the past 10 years maybe. My dad tends to remind me that back in school I didn’t like reading too much (perhaps because I didn’t usually need to work too hard to pass courses (except for math), or maybe I just hadn’t found my thing yet. Or maybe I was an immature brat? Perhaps. Anyway. What I’m reading right now is the Bridge Trilogy, by William Gibson. No big shocker here, I’ve read his works multiple times. I think this trilogy is the one I’ve read the least. That’s not to say it isn’t good, but it’s just gotten less attention from me. I’m on the final book now, ‘All tomorrow’s parties”. After that I’ll hop away from Gibson, and move on to James Bamford’s “The Shadow Factory”, a book on the NSA.

Since I misplaced (probably lent it out to someone who doesn’t remember or really liked the book) my copy of Stealing the Network – How to own a Shadow, I ordered a used copy from amazon. The condition was listed as very good, and it came exactly in that shape….

.. only it smells like weed. You know? Mary jane? Now it might just be from hemp-scented incense, or maybe just a pot-head security guy. I don’t mind really, but I still put the book outside for a while to get the worst fumes out. Luckily nobody had ripped pages to roll their joints in. I guess the book would then have been listed as.. Cannabilized. Get it!?!

 

Some notes from the road

First I want to talk a little bit about airport security. This was the first trip that I was ‘nude-scanned’. The scanner was at McCarran International Airport in Las Vegas. The device doing the scanning is a ProVision ATD. The type of scanner this is, is a millimeter-wave scanner. Unlike the X-ray backscatter type machines, these should not pose any health risks, as the radiation is not ionizing. There’s a comparison of the two technologies here.  When we flew in to the US, I saw the same machines deployed at O’Hare in Chicago, however, for some reason they were not being used. A regular metal detector was used instead. At LAS when leaving, they put some passangers through the millimeter-wave scanner, and some people through the metal detector. When it was my turn, four people had just passed through the metal detector. For no apparent reason (I didn’t notice a pattern), they closed the metal detector, and put me through the millimeter-wave scanner. You step into the device, and turn 90 degrees to face a set of instructions. There are spots on the floor marked for where your feet go, and you are instructed by a picture to hold your hands above your head. The device appears to do a sweep (looks like the door is closing on the round device), and then the TSA attendant asks you to step out.

Later, they also switched it around, bringing some people through the metal detector, and some through the scanner. Shit. Almost wrote scammer, there. An associate of mine walked through the scanner after me, and after that, he was patted down by the TSA agent. Why? Was he armed to the teeth? No, he was carrying a standard Finnish passport in his pocket. So the gorillion dollar device can’t distinguish between a passport and something that can be used as a weapon? Looking at some of the images of the user interface, and what I was able to see myself, the screen that they look at doesn’t show an image of a person when he or she is being scanned. Just a grey screen, which appeared to turn green when everything was okay. I didn’t see the “failed” scan, but i assume it might have shown the location of the suspected item. But, a passport? For reals? I felt a whole lot less secure after seeing that…

Also, how do they pick who gets the scanner, and who gets the metal detector?

A noteworthy detail is that there was a sheet of paper outside the machine which explained the technology, and the last row was something like “The use of this technology is optional”. Optional, when you’re four steps from the machine? I’m sure declining at that point would set off zero rectal search alarms? I was planning on declining myself, but I guess I might have chickened out / noticed the note a bit too late. I guess it would have meant the metal detector + a pat down, even if nothing beeped. And some gruntled TSA personel.

There were also new “rules and regulations” on the inbound flight. The Lufhansa flight attendants were ‘required’ to tell us that “grouping in the aisle or near the toilets or the kitchen during the flight is not permitted”. There was an incident on our Frankfurt -> O’Hare  747-400 type airplane where two people were using their phones near the toilets (both were of non-caucasian descent, if that matters), and the flight attendants announced, apparently due to this, that no loitering near the toilets was allowed. The two men declined to move, or didn’t listen, and a flight attendant was there very quickly, asking them to take their seats. After that, the captain turned on the fasten seat-belts sign. There was a rather clear connection between the two events. There was no turbulence, and we were thousands of miles from O’Hare.

You’re wondering about them using their phones? The flight had (paid) WLAN on it. The only caveat was that you were not allowed to use VOIP type applications, as per the terms of service that you accepted when you bought the service. 1 hour was 9,50 €, and 24 hours was 19,90. The connection was provided by satellite, and the service provider was T-mobile out of Germany. Latencies from the middle of the atlantic (or geostationary orbit i suppose?) to Finland were around 600-800 milliseconds. Downspeed was ~3Mbps, and up was 0.03 Mbps, according to Speedtest.net.

The connection worked very well in general, if you didn’t mind the slow upspeed, buying the service was easy with credit card. A notable detail is that when you associated with the AP, you had DNS resolution, so you could maybe have, contrary to the service agreement, have tunneled out using DNS, and something like NSTX. I didn’t poke around more, nor did I take too many other details off of the connection, but those are my notes.

I just had to edit this again to add this: God damn it it grinds my gears when people do not behave on aircraft. Jesus Herbert Christ! On the return flight to Helsinki, we had an awesome flight attendant. Funny, well spoken and approachable. When we were wheelsdown in Helsinki, still taxiing to our gate, a guy just decides to stand up to get his stuff. She told him “Sir, sit down!”. He did. Before we were completely stopped, and the captain had turned off the seat-belt light, there were at least ten mobile phones powering on, and seatbelts clicking lose.

People: You do not get off the airplane any faster by doing these dumb things. And if you tumble and fall, or drop some luggage on me while doing this shit, I will go medieval.

After we had come to a complete stop, I told the flight attendant “Same thing every time, huh?”, and she said “YEAH! Every time! Why do they keep thinking they’ll get off the plane faster?” She then turned to the man who had stood up during taxi, and asked him, “Sir, why did you stand up? Why? You could get seriously hurt!”, and he just shrugged and avoided her very piercing gaze, mumbling something under his breath.

P.P.S. Oh, and also, there are no bookshops on the Strip in Vegas. Just a hint to anyone who wants to maybe, I don’t know, make some money. I asked the concierge at the hotel we were staying at, and she said I’d probably have to get a cab to get to the nearest book store…

HTPC 2013

So about the HTPC…

It’s now 2013. Three years since I bought the thing, or so. It’s still running fine. I’ve done some upgrades during the past months, and I’ll discuss them in this article.

First of all, the CPU fan had to go. And by go, I mean replaced with a thin form factor, larger fan. The fan is attached to the case with some wires. It looks ugly, but then again, you don’t really see it from where you sit in the living room. There’s little to no vibration or noise from the fan. The one i got, and that i can recommend to any case that needs a low-RPM thin fan, was the Scythe Kaze Jyu Slim, from Jimm’s PC Store. I paid 8 euros for it. Works like a charm. It’s not attached to the CPU heatsink, but it still moves enough hot air out to keep things running. I guess I could run things passively, as I have speculated in the past, but I don’t really like my stuff running that hot, even if it’s within spec. Things just tend to last longer when they are at least somewhat cooled.

The second thing I replaced, was the hard disk. I wish I had a few extra bucks for an SSD, because that is what I will put in as an OS drive (if just for the fast boot time), but right now, I opted for a 3TB Western Digital Red. The old drive was a Western Digital Green 1TB, which had a number of issues (I lost one drive due to a feature relating to power saving, which wore out the drive prematurely. The warranty of course covered this, and no problems with WD). The drive also was a bit sluggish, it felt. But then again, the Green series drives are “supposed” to be. They run at lower RPMs, and are designed for power saving instead of high performance. The Red series drives (I paid around 150 for my 3TB version) are designed for NAS use, and are rated for a very large amount of usage hours. The HTPC is pretty much always on (well not really, but a lot of the time), so this was a good choice. I’ve now had it in use for a few months, and I can’t say I have any complaints. The drive runs smooth, silent and has a lot of capacity. It’s also, unsurprisingly, faster than the Green drive. I have a 20 GB partition set aside for the OS (which I will get to in the next paragraph) and the rest for media and backup from my desktop (over smb, I suppose it could be nfs too..). Nothing bad to say about the drive really. 3TB should be enough for everyone. “:)”

As for the OS, I am now running XBMC 12 “Frodo” RC2. There’s an RC3 but I have not upgraded, except for what I get through apt-get. I have to say that this is by far the best “out-of-the-box” XBMC experience so far. Every damn thing worked. The only thing I really had to set, was the audio output, since I’m running it out through SPDIF instead of HDMI (which was the default). I now no-longer had 9 audio devices to choose from (as in XBMC 11), but three. HDMI, SPDIF, and analog, which is exactly what you would expect. Before I had three devices, each with the three options. Very confusing. I also didn’t have to fiddle around with alsaconf or anything else to get both stereo and surround sound to work using the same output. Very much recommended.

Ok so here’s where I’m at right now. There are two final upgrades I would like to do, and I would like to finalize the fan-attachment so that it doesn’t look like ass. Still thinking about how to do that. The other two, are: SSD for the OS (I could take any size, really, as XBMC takes around 4 to 5GB), and upgrade the RAM from 2 to 4GB.