Building a NAS Server

After reading a review of the Drobo FS, I became obsessed with network attached storage (NAS). I realised that a NAS device would neatly solve a couple of long-standing problems I hadn’t got around to fixing: data backup and data organisation.

This post will explain how I picked the hardware and software for my NAS.

To buy or to build?

The Drobo FS itself, while a compelling product, is expensive. There are also some worrying stories of problems with poor read/write speeds, noise, and the slightly ropey client software. But mainly, I feel it’s my duty to build my own computers.

Nevertheless, maybe you run a small business and prefer to think of a NAS as an appliance – something that has a warranty and a customer support email address. This is where the pre-build solutions from Drobo, Synology, ReadyNAS, etc., excel. SmallNetBuilder is a great resource for finding out more about that sort of thing.

Hardware

I guessed that 4 x 2TB hard drives with single-drive redundancy would suit me, leaving a usable space of around 5.5TB. With that settled, I could pick the hardware:

• CFI A7879 Mini-ITX NAS Server Case (£98.40 from LinITX)
• Atom 330 CPU (dual-core) + ASUS AT3IONT-I Mini-ITX motherboard + fanless heatsink (£112.83)
• 4 x 2TB hard drives (£240 total)
• 2GB memory (£19.99)
• 2 extra SATA cables (£3.65 total)
• 2GB USB boot drive (pretty much free)

…totalling £474.87 (about 770USD). This is almost exactly the cost of a 5-bay Drobo FS, without drives, at £476.54!

Case

The case was a little expensive, but it does the job well. Some basic instructions would have been nice – it took 5-10 minutes of fiddling to figure out how to get drives inside. The other 4-bay NAS case option I found, the Chenbro ES34069, was even more expensive. (For a 6-bay NAS, the Lian Li PC-Q08 got a decent review, but the Fractal Array R2 did not.)

CPU and motherboard

I chose the motherboard because it hit the sweet spot of price, power consumption, number of SATA ports, and availability. Also, FreeBSD supports the Ethernet chipset, which was important (see below). Newer Atom CPUs, such as the D510, should be more power-efficient overall because of the on-die memory controller and graphics, but none were readily available.

Temperature, noise, and power

I wanted to minimise noise, so passive CPU cooling appealed to me. The case includes a PSU fan and a 120mm case fan, which are fairly quiet. The motherboard does dynamically adjust the case fan speed based on the CPU temperature. CPU and hard disk temperatures are normally 35-45°C, depending on the ambient temperature. The hard disks can be noisy when seeking.

The normal total power consumption is about 43W. This is while running a BitTorrent client, and so all four hard disks are spinning. I’m fairly happy with this, because the drives are benchmarked at about 4W idle, and 5.6W while seeking. Because the NAS will be running 24 hours a day, it should cost about £43 a year to run (1W roughly equates to £1 over a year).

If I turn BitTorrent off and allow all the drives to power-down to standby mode, power consumption drops to about 29W. For comparison, the Drobo FS is rated at about 12W (idle) and 56W (busy), with four drives.

OS/software

Picking hardware was time consuming, but relatively painless. Software was a lot trickier to nail down, given all the options:

1. FreeNAS, using ZFS
2. Plain FreeBSD, using ZFS
3. Solaris, using ZFS
4. Linux and mdraid (software RAID)
5. Linux and ZFS, via ZFS-FUSE
6. unRAID
7. OpenFiler
8. Greyhole
9. NexentaStor
10. OpenIndiana
11. Windows Home Server (2007 is often preferred to 2011 because of the Drobo-like Drive Extender)

The only easy part was that I wouldn’t settle for anything less than ZFS. (The next section explains why.) This eliminated Linux (I didn’t want ZFS-FUSE because of the performance implications) and OpenFiler. I also ruled out unRAID, NexentaStor, and Windows Home Server, because they (can) cost money. Greyhole looks like a great idea (similar to ditto blocks in ZFS), but I didn’t want to have to manually split my data into “important” and “unimportant” bins. Also, parity-based redundancy is far more efficient, in terms of hard drive space, than the full mirroring offered by Greyhole – especially as more drives are added.

Why ZFS?

“End the suffering” — Jeff Bonwick & Bill Moore

ZFS originally stood for Zettabyte File System (due to the immense storage limit of 256 x 10¹⁵ zettabytes), but it is much more than just a file system. Importantly, it is also a kind of logical volume manager (LVM). That means that ZFS likes to control both the allocation of physical drives to logical storage entities (vdevs and zpools in ZFS), as well as the organisation of files within those zpools. This gives ZFS significant power, compared to a layered approach.

For a NAS device, a couple of features are worth looking at in more detail: data integrity and RAID-Z.

Self-healing data

ZFS really cares about keeping your data safe. CERN – which famously handles a lot of data – did an experiment with 3000 hardware RAID cards. They experienced 152 cases of silent data corruption within three weeks. Although slightly mystical-sounding, bit-rot really can happen.

In order to provide the so-called end-to-end data integrity, ZFS simply stores checksums away from data blocks. In fact, the checksums themselves have their own checksums further up the chain, and so on until the root of the file system (known as the uberblock). The entire file system hierarchy is a self-healing hash-tree. Here’s a simplified diagram:

Going from left to right, directory 1 contains a pointer to file A, file A’s checksum, and some other meta-data. But dir 1 is just another data block, which is pointed to by the uberblock. The uberblock therefore contains a checksum of dir 1, and so on. This does mean that every write to a file involves recalculating several checksums, all the way back to the root node (the uberblock). But ZFS’s copy-on-write policy (see below) and transactional nature mitigate the performance penalty. Furthermore, ZFS is designed to take advantage of Moore’s law: CPU cycles are cheap, but hard drives are slow.

This all means that ZFS can check your data at any time — and it can often repair data after a variety of problems.

RAID-Z: fixing RAID 5

ZFS’s answer to RAID 5 is called RAID-Z1, but it has important advantages over hardware RAID 5. RAID-Z avoids the dreaded write hole, where loss of power during a write operation can (silently!) leave the data drive and parity drive inconsistent. In ZFS, the copy-on-write policy solves this problem because ZFS always writes to a new block, and then atomically updates the block pointers:

A copy of the file system structure is made when in the process of writing new blocks. If a drive failure happens during the write, the original data is still accessible and the file system knows that the write operation didn’t complete.

RAID 5 also has a performance problem with partial stripe writes, because the old stripe data must be read before being modified and written back. RAID-Z uses dynamic stripe width, so all writes are full-stripe writes and this problem can never occur.

FreeNAS

What is it, and why do I need it?

FreeNAS is based on FreeBSD, but with a few clever changes that make it great for NAS servers. Naturally, it inherits FreeBSD’s ZFS support, if a little out of date. I decided to use FreeNAS, rather than any other ZFS-capable OS, for two main reasons:

1. The boot-from-scratch philosophy makes backing-up and restoring the entire OS (well, the XML settings file, not a binary image) very simple. If anything ever goes badly wrong with the OS, I just burn the OS image to a USB drive and upload the settings file. Nothing to install, nothing to configure. It also means that the OS is not stored on a data drive. And booting from USB ensures that the OS does not use up a valuable SATA port. (In fact, FreeNAS boots from USB and copies the OS to a RAM-disk for two reasons: 1) the OS is “clean” at each boot, as described above, and 2) to minimise writes to the flash memory.)
2. FreeNAS’s purpose-built web interface means that you rarely need to ssh into the server and use commands to get stuff done. In fact, once it’s up and running, you rarely need to touch anything.

Lots of extras

FreeNAS should have everything you need from a NAS server:

• Set up SMB/CIFS, FTP, NFS, and AFP shares
• Transmission BitTorrent client, with its own web interface (24-hour downloading/seeding!)
• rsync
• SSH
• iSCSI
• Dynamic DNS
• DAAP
• UPnP
• S.M.A.R.T. monitoring and email alerts
• Manage users
• Network interface link aggregation

Version 8?

There was one last decision. Version 8 is a complete rewrite of version 0.7, but many useful features are still missing. Also, I couldn’t write a bootable USB image for 8 (although the CD image worked ok in VirtualBox). For now, 0.7 is the only choice.

Performance

Who cares about how elegant the OS and file system are; more importantly, how fast is it? Well, not great, but not bad. Without any significant tuning, and using SMB shares, I get 20-30MB/s for sustained reads and writes, over gigabit Ethernet:

Sadly, it seems that the RealTek NIC (or FreeNAS’s driver) doesn’t support jumbo frames; the MTU setting in FreeNAS won’t budge over 1500 bytes. Wireshark confirmed that no packets exceeded 1500 bytes. But, there is still some tweaking to be done…

What now?

A NAS server is a huge step forward from the horrors of USB external hard drives (or, further back in time, burning DVDs and CDs…) for backups. Aside from filling the NAS with 5.18TB of data, there are minor improvements to look at:

• Setting up email notification for hard drive S.M.A.R.T. attribute thresholds.
• Power and temperature improvements form underclocking the CPU, and ensuring that the GPU is powered-down.
• Improving read and write speeds or, at least, pin-pointing the performance bottleneck.

Other practical notes

• It’s important to note that RAID-Z1 (single-drive redundancy) is not really recommended. If a single drive fails, the strain of resilvering, which may take a long time, might knock out a second drive. All data would be lost. RAID-Z2 provides dual-disk redundancy for this reason. (All of my work and important personal documents are backed-up with Dropbox; the data on my NAS are somewhat expendable.)
• Although Solaris offers the most up-to-date ZFS support, with features such as de-duplication and snapshots, I didn’t plan on needing these and so I was happy to settle for the older version of ZFS in FreeNAS 0.7.
• The SATA ports in the case are close together, and I was lucky to only have two L-shaped SATA cable connectors; otherwise they would not fit.
• The case also has a 2.5″ hard drive bracket, which is intended to be used for the OS/boot drive.
• Booting from USB automatically with the ASUS AT3IONT-I motherboard is not obvious. You need to pretend the USB drive is a floppy disk drive.
• Initially, I toyed with the idea of an integrated NAS and HTPC, but decided that there would be too many compromises; dedicated devices would be far better. For example, FreeNAS is great for ZFS support and other NAS tools, but how would I get a driver for ION-accelerated HD video out of the HDMI port? The hard drives are also too noisy for a HTPC. As an aside, the Boxee Box works really nicely as a dedicated HTPC.
• btrfs should be great one day when it’s ready, but ZFS does the stuff that’s important for a NAS now.

Further reading of ZFS and NAS building

• Ten Ways To Easily Improve Oracle Solaris ZFS Filesystem Performance
• A Home Fileserver using ZFS
• Build Your Own NAS Device
• DIY: Home NAS Box with OpenSolaris and ZFS
• AnandTech — File Server Builder’s Guide