Home Server/Setting up your Storage

Setting up your RAID Solution

Note: Before Reading, if you do not have a grasp on basic RAID concepts you should read up on that first. There is terminology in this article you will not understand without first understanding RAID.

ZFS

Note: There are a lot of misconceptions about ZFS and ECC Ram. ECC Ram is NOT required for ZFS to operate. ZFS was made to protect data against degradation however, and not using ECC Ram to protect against memory errors (and thus data degradation) defeats the purpose of ZFS.

Basic Concepts

Adaptive Replacement Cache (ARC)

Physical Disks are grouped into Virtual devices (Vdevs). Vdevs are grouped into Zpools. Datasets reside in Zpools.

The actual file system portion of ZFS is a dataset which sits on top of the ZPool. This is where you store all of your data. There are also Zvols which are the equivalent of block devices (or LVM LVs). You can format these with other file systems like XFS, or use them as block storage, but for the most part we will be using just the standard ZFS file system. There are also Snapshots and Clones which we will talk about later.

Zpools stripe data across all included vdevs.

There are 7 types of vdevs.

1. Disk: A single storage device. Adding multiple drives to the same pool without RAIDZ or Mirror is effectively Raid 0.
2. Mirror: Same as a RAID 1 Mirror. Adding multiple mirrored vdevs is effectively Raid 10.
3. RAIDZ: Parity based RAID similar to RAID 5. RAIDZ1, RAIDZ2, RAIDZ3 with Single, double, and triple parity respectively.
4. Hot Spare: A hot spare, or standby drive that will replace a failed disk until it is replaced with a new one.
5. File: A pre-allocated file.
6. Cache: A cache device (typically SSD) for L2ARC. It's generally not recommend to use this unless you absolutely need it.
7. Log: Dedicated ZFS Intent Log (ZIL) device, also called a SLOG (Separate intent LOG SSD). Usually these are high performance, durable SLC or MLC SSDs

Tip: When setting up your drives in ZFS, instead of using the raw disks, make fixed size partitions on each, a little smaller than your drive's full capacity and use that instead. Not all disks have exactly the same sector count. Later down the line if you buy a new drive to replace a failed disk that's a different model or manufacturer, even if it's the correct capacity it may have less sectors and ZFS will not let you use it.

Features

• Copy on Write.

• Drive pooling

• Snapshots and Clones.

• Checksums for data integrity.

• Automatic File Self Healing.

• Compression (uses Significant CPU resources)

• Deduplication (uses Significant CPU resources)

Requirements

All those features come with requirements.

As mentioned above, it is *highly* recommended to use ECC ram with ZFS. This means you should NOT use an SBC, consoomer computer, or shitty NAS like QNAP or synology. This is a major limitation of ZFS, but you don’t necessarily need a 4U monster rack mount to get it done. Workstation-grade xeons that can be had for cheap (used) on eBay with ECC ram that work great. They look like a normal desktop tower and are silent. Even something like a i3 9300 on a server motherboard will be enough to run most of the features (would not recommend compression and Dedup).

Your drives need to be exposed to the operating system DIRECTLY, that means if you have a hardware raid card, it has to be set to “IT MODE” or “JBOD” (just a bunch of disks). Not all raid cards support this, so research before you buy. If you are planning on running ZFS in a virtualized environment you need to pass your SAS HBA card to the VM to allow it to interact with the disks. If de-duplication is very important to you, you will need a lot of ram - 1GB per TB is the rule of thumb tossed around a lot. If you do not need de-duplication, like most people don’t, the ram requirements are reduced.

Should I use ZFS?

ZFS has a lot of really great features that make a a superb file system. It has file system level checksums for data integrity, file self healing which can correct silent disk errors, copy on write, incremental snapshots and rollback, file deduplication, encryption, and more.

There are however, some downsides to ZFS RAIDZ. Notably inflexibility and the upfront cost. ZFS RAIDZ vdevs CANNOT BE EXPANDED after being created. Parity cannot be added either (you cannot change a RAIDZ1 to a RAIDZ2 later on). You cannot use differently sized disks or disks with data already on them (even disks formatted as ZFS). In other words, you need to buy ALL of the drives you plan on using in your RAIDZ array at the same time, because unlike other software RAID (or even hardware RAID), you won't be able to change it later. This inherently requires you to pre-plan your expansion. It is best to budget your hard drive money and save for the major sales and buy multiple shuckable external hard drives at once. This means you get maximum TB/$ spent and all the benefits of ZFS RAIDZ.

Note that the above limitations and pre-planning are only required if you want to have raid-like features. It is perfectly fine to use ZFS on a single drive and using snapshots + ZFS send/receive makes block-level perfect backups to a second drive (that can be different sized) while using minimal space, protect from ransomeware, and are faster than any rsync or a cp copy. Remember that ‘”RAID in any form is NOT a backup”’ - RAID provides uptime and availability in case of a drive failure. So when asking yourself "Should I use ZFS RAIDZ?” you really should be asking "Do I really need RAID” (Is uptime and availability in case of drive failure very important to me?) and "Can I budget/plan my purchases around buying multiple drives at once?”. If the answer to both of those questions is "Yes", then you can and should use ZFS RAIDZ. If no, but you still want uptime, use something else like Snapraid or mdadm. If no, but you still want all the features of ZFS and meet the requirements listed above, then use ZFS on single drives.

General Recommendations

Not Recommended:

• Running ZFS on ancient hardware.
• Running ZFS on consoomer motherboards.
• Run ZFS without ECC Ram. If you can afford ZFS you can afford to get ECC Ram. No excuses.
• Run ZFS on underqualified hardware (shitty little NAS boxes, SBCs, etc).
• Use "Mutt" pools (Zpools with differently sized vdevs).
• Growing your Zpool by replacing disks. Backup your data elsewhere, create a new pool, and transfer the data to the new pool. Much faster. (You could theoretically use a USB drive dock provided your array is 5 disks or less).

DO NOT

• Run ZFS on top of Hardware RAID.
• Run ZFS on top of other soft RAID.
• Run ZFS in a VM without taking the proper precautions.
• Run ZFS with SMR drives.

DO

• Run ZFS if you have ECC ram and a sandy-bridge or newer processor
• Run ZFS if you care about having the best and most robust set of features in any file system
• Use snapshots (see syncoid, snapoid, and other handy tools to schedule and manage snapshots)
• Use ZFS send/receive for backups
• Sleep soundly with a smile on your face knowing that you have the most based filesystem in the world

Hardware RAID

Warning: Using hardware RAID and software RAID at the same time is NOT recommended. If you wish to pool multiple hardware RAID arrays together into a logical volume use LVM.

If you bought an old used server with a RAID controller already installed, or perhaps you don't feel like messing with software RAID solutions, you have the option of using hardware RAID rather than software RAID.

Choosing a file system

NTFS

If you are using Snapraid as your raid solution, using NTFS formatted drives is perfectly fine. With Snapraid you are usually pulling out random drives you have lying around, which are most likely to be NTFS formatted. Otherwise, we do not recommend using NTFS unless you are running a Windows server for some reason. It does not have the same level of support on Linux and UNIX based systems as ext4 and XFS.

unRAID does not support NTFS. If you are using unRAID you will need to use ext4 or XFS.

Pooling together your storage devices

Graphic overview of drive pooling options. Note: Only LVM and ZFS are mutually exclusive. You can use mergerfs on top of ZFS or filesystems formatted on LVM LVs.

Pooling drives together into logical volumes saves you the hassle of having to spread your data across many physical drives, especially as your individual directories outgrow your physical disks. Instead, you can add your drives to a single logical volume that acts just like a normal volume does only it can be much bigger than any single disk, and more flexible too. ZFS already does this inherently with Zpools, but there are other options available to pool multiple different drives together.

LVM

Warning: Using LVM without any form of redundancy is not recommended. LVM distributes data across all member drives. This is effectively a RAID 0, losing a disk will result in significant data loss.

External Links

• ZFS Documentation
• FreeBSD handbook: ZFS'
• Arch wiki: ZFS

See also

• Home server
• Storage devices