Pi musings

So now I’ve gone and done it! I am doing something with my Pi. What I’ve done is, install nginx in a jail on it. Why? Just because I haven’t done that before. I’ll talk a bit more about what I did, and how in this post.

Why nginx? Well, the primary reason is that it’s growing in market share, and because I have very little hands-on experience of it. Also because I have this idea in my head that it’s slightly less bulky than say Apache2. Many Pi-specific pages also recommend lighthttpd, but since nginx is more prevalent on the net, I chose that.

Note! You could prepare the chroot environment beforehand. If you wish to do so, jump to the appropriate heading and then come back here. This is the order that I did things in, so if you, for some yahoo reason want to follow that, read on.

The Raspbian repositories contain a version of nginx, but it’s supposedly very old. I opted to compile from source, which seemed like a good idea after the repositories listed for a more current version didn’t work properly for the version of Raspbian / architechture of the Pi. Obviously, compiling on the Pi as a rather slow process, but this isn’t a rush order. To start off, i installed some necessary tools so I could compile from source:

sudo apt-get -y install wget build-essential libpcre3-dev libpcre++-dev zlib1g-dev libssl-dev

After this, wget the latest source package for ngingx, http://nginx.org/en/download.html, and unpack this to a location of your choosing:

wget http://nginx.org/download/nginx-1.5.6.tar.gz and the pgp signature: wget http://nginx.org/download/nginx-1.5.6.tar.gz.asc

Get the public key for the signer of the package (in t his case Maxim Dounin)  wget http://nginx.org/keys/mdounin.key

Import it: gpg –import mdounin.key

And finally run gpg nginx-1.5.6.tar.gz.acs

You should get a message about a good signature, however, it’ll not be a trusted signature. You can’t be sure it belongs to the owner. The key would need to be signed by trusted sources, in order to establish the web of trust properly. But for now, we are content.

Then once you are all wrapped in tin foil, go prepare a pot of your favorite coffee and start compiling nginx. Change, add, remove options as needed. This is just from another howto, so you might like different locations for your logs, or include modules that are not included here:

cd nginx-$VERSION ./configure –sbin-path=/usr/sbin/nginx \ –conf-path=/etc/nginx/nginx.conf \ –pid-path=/var/run/nginx.pid \ –error-log-path=/var/log/nginx/error.log \ –http-log-path=/var/log/nginx/access.log \ –with-http_ssl_module \ –without-http_proxy_module make

After this, you could potentially start nginx using /usr/sbin/nginx, but we’re not done yet.

Chroot

Here, we want to do some potential damage control. The webserver is living inside its own little world, and if someone gets into that world, it’s kind of small and boring, and has no real access to the underlying OS.

We can do this either manually, or by giving the chroot directory (the new root) as a variable:

D=/example
mkdir $D

After this, we need to create necessary directories inside the chroot directory for nginx to work properly.

# mkdir -p $D/etc
# mkdir -p $D/dev
# mkdir -p $D/var
# mkdir -p $D/usr
# mkdir -p $D/usr/local/nginx
# mkdir -p $D/tmp
# chmod 1777 $D/tmp
# mkdir -p $D/var/tmp
# chmod 1777 $D/var/tmp
# mkdir -p $D/lib

Note that we also give permissions to tmp and /var/tmp at this stage. Just to keep them writable by everyone just like they are in the base OS. Makes it easier for non-privileged users to write temporary files during installs or stuff needed when you are running the server.  Some instructions (like the one on Nixcraft that I relied on heavily while doing this) create a lib64 directory inside the chroot. I didn’t even have such a directory in the base Raspbian, so I followed suite inside the chroot by making a lib directory.

Next, create the following inside the chroot/dev directory, but first checking their special attributes using:

# ls -l /dev/{null,random,urandom}

You’ll get something like:

crw-rw-rw- 1 root root 1, 3 Jan  1  1970 /dev/null
crw-rw-rw- 1 root root 1, 8 Jan  1  1970 /dev/random
crw-rw-rw- 1 root root 1, 9 Jan  1  1970 /dev/urandom

Note column five. 1,3 and 1,8 and 1,9.  You need to set these attributes inside the chroot too. Do a:

# /bin/mknod -m 0666 $D/dev/null c 1 3
# /bin/mknod -m 0666 $D/dev/random c 1 8
# /bin/mknod -m 0444 $D/dev/urandom c 1 9

Next, you’ll copy all the nginx files from your base OS inside the chroot. For instance:

# /bin/cp -farv /usr/local/nginx/* $D/usr/local/nginx and

# /bin/cp – farv /etc/nginx/* $D/etc/nginx

Next a tricker part. Move all necessary libraries to run nginx to the chroot. You can find out what you need by doing a:

ldd /usr/sbin/nginx

You’ll get an output similar to:

/usr/lib/arm-linux-gnueabihf/libcofi_rpi.so (0xb6f94000)
libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6f6a000)
libcrypt.so.1 => /lib/arm-linux-gnueabihf/libcrypt.so.1 (0xb6f33000)
libpcre.so.3 => /lib/arm-linux-gnueabihf/libpcre.so.3 (0xb6ef2000)
libssl.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libssl.so.1.0.0 (0xb6ea2000)
libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0 (0xb6d3f000)
libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6d34000)
libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xb6d16000)
libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6cee000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6bbf000)
/lib/ld-linux-armhf.so.3 (0xb6fa1000)

All of these need to go to the corresponding locations inside the chroot. There are scripts floating around for checking what you need and copying them over; I just copied them manually because I’m a pleb.  You can always come back later; nginx and any other tools you use will tell you if you uare missing any libraries, and you can copy them later.

Copy the relevant contents of /etc to the chroot. I had problems with the users inside the chroot, but it might have been something I messed up. I was unable to run it using nobody:nogroup, and had to resort to using the uid and gid, but more on that later. If someone knows what I fucked up, and happens to read this, use the comments, thanks! But the copying I mentioned (again thanks to Nixcraft):

# cp -fv /etc/{group,prelink.cache,services,adjtime,shells,gshadow,shadow,hosts.deny,localtime,nsswitch.conf,nscd.conf,prelink.conf,protocols,hosts,passwd,ld.so.cache,ld.so.conf,resolv.conf,host.conf} $D/etc

And some directories (though my raspbian install didn’t have prelink.conf.d at all):

# cp -avr /etc/{ld.so.conf.d,prelink.conf.d} $D/etc

We’re just about done. Kill an existing nginx’s using pkill nginx or something like killall -9 nginx to do it more violently.  Then we can run a test of nginx inside the chroot. This will tell you what is missing (libraries, files etc.), or if your config syntax is wrong:

# /usr/sbin/chroot /nginx /usr/local/nginx/sbin/nginx -t

To run it finally, remove the -t at the end. As I mentioned, at this point I had issues about a line in the nginx config file (/etc/nginx/nginx.conf), which is “user nobody;”. For the life of me  I could not get it to run using this user, even though I had it inside the chroot/etc/passwd, and group files. It just told me unknown user and so on. Changing the user also had no effect, i tried creating a fresh user, but to no avail. Finally, I ended up running nginx with:

/usr/sbin/chroot –userspec=65534:65534 /nginx /usr/sbin/nginx

Where 65534 is the uid and gid (respectively) of nobody and nogroup. Note that we are chrooting into /nginx (my chroot directory for nginx) and then from there, running /usr/sbin/nginx which is the script that starts nginx. After this, we have nginx running under the correct user and group:

nobody    4355  0.0  0.1   4984   724 ?        Ss   Oct07   0:00 nginx: master process /usr/sbin/nginx
nobody    4356  0.0  0.2   5140  1228 ?        S    Oct07   0:00 nginx: worker process

To be absolutely sure that nobody runs the “base OS” version of nginx, you can remove the directories associated, or rename the executable file under /usr/sbin (i called mine nginx_nonchroot), so I can verify that file isn’t being run. Or remove the execute bit with chmod -x /usr/sbin/nginx.

When starting nginx at boot, be sure you are doing it in the right way to ensure it’s inside the chroot:

# echo '/usr/sbin/chroot /nginx /usr/sbin/nginx' >> /etc/rc.local

To verify that your nginx is running inside the chroot, use the process id (second column when you run ps aux | grep nginx; in my example, 4355), by running:

# ls -la /proc/4355/root/

…and you’re getting the contents of the chroot root, i.e. all the directories that sit under the chroot /

drwxr-xr-x 10 root root 4096 Oct  7 19:00 .
drwxr-xr-x 24 root root 4096 Oct  6 23:24 ..
drwxr-xr-x  2 root root 4096 Oct  7 19:11 bin
drwxr-xr-x  2 root root 4096 Oct  6 23:25 dev
drwxr-xr-x  5 root root 4096 Oct  7 19:43 etc
drwxr-xr-x  3 root root 4096 Oct  6 23:36 lib
drwxr-xr-x  2 root root 4096 Oct  7 00:03 run
drwxrwxrwt  2 root root 4096 Oct  6 23:23 tmp
drwxr-xr-x  5 root root 4096 Oct  6 23:27 usr
drwxr-xr-x  5 root root 4096 Oct  7 19:51 var

You can also change the default index page so you can see that that’s the one being loaded.  In my case /nginx/usr/local/nginx/html/index.html. You can reload the chrooted nginx using:

# /usr/sbin/chroot /nginx /nginx/usr/sbin/nginx -s reload

You could now make sure nginx is listening on your pi, by using:

netstat -pantu | grep nginx

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4355/nginx   

Browse to the ip assigned to your pi and see your webpage! Make sure you lock things down with iptables, and allow traffic only to ports that you want, and from addresses you want.

Infinite props to Nixcraft for this article, which helped me along the way. The main reason I wrote this was that my install  was slightly different, and I figure I’d type my own problems and solutions down. Also, raspbian has changed slightly (i guess?); So here you are. This howto was also very helpful, thanks to elinux.org.