We are still actively working on the spam issue.
Difference between revisions of "Home Server/Setting up your Storage"
(Minor edits and additions) |
|||
| Line 36: | Line 36: | ||
{{Tip|When setting up your drives in ZFS, instead of using the raw disks, make fixed size partitions on each, a little smaller than your drive's full capacity and use that instead. | {{Tip|When setting up your drives in ZFS, instead of using the raw disks, make fixed size partitions on each, a little smaller than your drive's full capacity and use that instead. | ||
Not all disks have exactly the same sector count. Later down the line if you buy a new drive to replace a failed disk that's a different model or manufacturer, even if it's the correct capacity it may have less sectors and [http://www.freebsddiary.org/zfs-with-gpart.php ZFS will not let you use it.]}} | Not all disks have exactly the same sector count. Later down the line if you buy a new drive to replace a failed disk that's a different model or manufacturer, even if it's the correct capacity it may have less sectors and [http://www.freebsddiary.org/zfs-with-gpart.php ZFS will not let you use it.]}} | ||
| + | |||
| + | ==== Features ==== | ||
| + | |||
| + | *Copy on Write. | ||
| + | |||
| + | *Snapshots and Clones. | ||
| + | |||
| + | *Checksums for data integrity. | ||
| + | |||
| + | *Automatic File Self Healing. | ||
| + | |||
| + | *Compression (uses Significant CPU resources) | ||
| + | |||
| + | *Deduplication (uses Significant CPU resources) | ||
| + | |||
==== Requirements ==== | ==== Requirements ==== | ||
| − | All those | + | All those features come with requirements. |
| − | As mentioned above, it is *highly* recommended to use ECC ram with ZFS. This means you should NOT use an SBC, consoomer computer, or shitty NAS like QNAP or synology | + | As mentioned above, it is *highly* recommended to use ECC ram with ZFS. This means you should NOT use an SBC, consoomer computer, or shitty NAS like QNAP or synology. This is a major limitation of ZFS, but you don’t necessarily need a 4U monster rack mount to get it done. Workstation-grade xeons that can be had for cheap (used) on eBay with ECC ram that work great. They look like a normal desktop tower and are silent. Even something like a i3 9300 on a server motherboard will be enough to run most of the features (would not recommend compression and Dedup). |
| − | Your drives need to be exposed to the operating system DIRECTLY, that means if you have a hardware raid card, it has to be set to “IT MODE” or “JBOD” (just a bunch of disks). | + | Your drives need to be exposed to the operating system DIRECTLY, that means if you have a hardware raid card, it has to be set to “IT MODE” or “JBOD” (just a bunch of disks). Not all raid cards support this, so research before you buy. If you are planning on running ZFS in a virtualized environment you need to pass your SAS HBA card to the VM to allow it to interact with the disks. If de-duplication is very important to you, you will need a lot of ram - 1GB per TB is the rule of thumb tossed around a lot. If you do not need de-duplication, like most people don’t, the ram requirements are reduced. |
| − | |||
| − | |||
| Line 54: | Line 67: | ||
There are however, some downsides to ZFS RAIDZ. Notably inflexibility and the upfront cost. ZFS RAIDZ vdevs '''[https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html CANNOT BE EXPANDED]''' after being created. Parity cannot be added either (you cannot change a RAIDZ1 to a RAIDZ2 later on). You cannot use differently sized disks or disks with data already on them (even disks formatted as ZFS). In other words, you need to buy ''ALL'' of the drives you plan on using in your RAIDZ array '''at the same time''', because unlike other software RAID (or even hardware RAID), you won't be able to change it later. This inherently requires you to pre-plan your expansion. It is best to budget your hard drive money and save for the major sales and buy multiple shuckable external hard drives at once. This means you get maximum TB/$ spent and all the benefits of ZFS RAIDZ. | There are however, some downsides to ZFS RAIDZ. Notably inflexibility and the upfront cost. ZFS RAIDZ vdevs '''[https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html CANNOT BE EXPANDED]''' after being created. Parity cannot be added either (you cannot change a RAIDZ1 to a RAIDZ2 later on). You cannot use differently sized disks or disks with data already on them (even disks formatted as ZFS). In other words, you need to buy ''ALL'' of the drives you plan on using in your RAIDZ array '''at the same time''', because unlike other software RAID (or even hardware RAID), you won't be able to change it later. This inherently requires you to pre-plan your expansion. It is best to budget your hard drive money and save for the major sales and buy multiple shuckable external hard drives at once. This means you get maximum TB/$ spent and all the benefits of ZFS RAIDZ. | ||
| − | Note that the above limitations and pre-planning are only required if you want to have raid-like features. | + | Note that the above limitations and pre-planning are only required if you want to have raid-like features. It is perfectly fine to use ZFS on a single drive and using snapshots + ZFS send/receive makes block-level perfect backups to a second drive (that can be different sized) while using minimal space, protect from ransomeware, and are faster than any rsync or a cp copy. Remember that ‘”[https://www.raidisnotabackup.com RAID in any form is NOT a backup]”’ - RAID provides uptime and availability in case of a drive failure. So when asking yourself "Should I use ZFS RAIDZ?” you really should be asking "Do I really need RAID” (Is uptime and availability in case of drive failure very important to me?) and "Can I budget/plan my purchases around buying multiple drives at once?”. If the answer to both of those questions is "Yes", then you can and should use ZFS RAIDZ. If no, but you still want uptime, use something else like Snapraid or mdadm. If no, but you still want all the features of ZFS and meet the requirements listed above, then use ZFS on single drives. |
| + | |||
| + | ====General Reccomendations==== | ||
'''Not Recommended:''' | '''Not Recommended:''' | ||
| Line 65: | Line 80: | ||
*Use "Mutt" pools (Zpools with differently sized vdevs). | *Use "Mutt" pools (Zpools with differently sized vdevs). | ||
*Growing your Zpool by replacing disks. Backup your data elsewhere, create a new pool, and transfer the data to the new pool. Much faster. (You could theoretically use a USB drive dock provided your array is 5 disks or less). | *Growing your Zpool by replacing disks. Backup your data elsewhere, create a new pool, and transfer the data to the new pool. Much faster. (You could theoretically use a USB drive dock provided your array is 5 disks or less). | ||
| − | |||
'''<span style="color: red;>DO NOT</span>''' | '''<span style="color: red;>DO NOT</span>''' | ||
Revision as of 20:54, 23 December 2020
Contents
Setting up your RAID Solution
mdadm
ZFS
Basic Concepts
Adaptive Replacement Cache (ARC)
Physical Disks are grouped into Virtual devices (Vdevs). Vdevs are grouped into Zpools. Datasets reside in Zpools.
The actual file system portion of ZFS is a dataset which sits on top of the ZPool. This is where you store all of your data. There are also Zvols which are the equivalent of block devices (or LVM LVs). You can format these with other file systems like XFS, or use them as block storage, but for the most part we will be using just the standard ZFS file system. There are also Snapshots and Clones which we will talk about later.
Zpools stripe data across all included vdevs.
There are 7 types of vdevs.
- Disk: A single storage device. Adding multiple drives to the same pool without RAIDZ or Mirror is effectively Raid 0.
- Mirror: Same as a RAID 1 Mirror. Adding multiple mirrored vdevs is effectively Raid 10.
- RAIDZ: Parity based RAID similar to RAID 5. RAIDZ1, RAIDZ2, RAIDZ3 with Single, double, and triple parity respectively.
- Hot Spare: A hot spare, or standby drive that will replace a failed disk until it is replaced with a new one.
- File: A pre-allocated file.
- Cache: A cache device (typically SSD) for L2ARC. It's generally not recommend to use this unless you absolutely need it.
- Log: Dedicated ZFS Intent Log (ZIL) device, also called a SLOG (Separate intent LOG SSD). Usually these are high performance, durable SLC or MLC SSDs
Features
- Copy on Write.
- Snapshots and Clones.
- Checksums for data integrity.
- Automatic File Self Healing.
- Compression (uses Significant CPU resources)
- Deduplication (uses Significant CPU resources)
Requirements
All those features come with requirements.
As mentioned above, it is *highly* recommended to use ECC ram with ZFS. This means you should NOT use an SBC, consoomer computer, or shitty NAS like QNAP or synology. This is a major limitation of ZFS, but you don’t necessarily need a 4U monster rack mount to get it done. Workstation-grade xeons that can be had for cheap (used) on eBay with ECC ram that work great. They look like a normal desktop tower and are silent. Even something like a i3 9300 on a server motherboard will be enough to run most of the features (would not recommend compression and Dedup).
Your drives need to be exposed to the operating system DIRECTLY, that means if you have a hardware raid card, it has to be set to “IT MODE” or “JBOD” (just a bunch of disks). Not all raid cards support this, so research before you buy. If you are planning on running ZFS in a virtualized environment you need to pass your SAS HBA card to the VM to allow it to interact with the disks. If de-duplication is very important to you, you will need a lot of ram - 1GB per TB is the rule of thumb tossed around a lot. If you do not need de-duplication, like most people don’t, the ram requirements are reduced.
Should I use ZFS?
ZFS has a lot of really great features that make a a superb file system. It has file system level checksums for data integrity, file self healing which can correct silent disk errors, copy on write, incremental snapshots and rollback, file deduplication, encryption, and more.
There are however, some downsides to ZFS RAIDZ. Notably inflexibility and the upfront cost. ZFS RAIDZ vdevs CANNOT BE EXPANDED after being created. Parity cannot be added either (you cannot change a RAIDZ1 to a RAIDZ2 later on). You cannot use differently sized disks or disks with data already on them (even disks formatted as ZFS). In other words, you need to buy ALL of the drives you plan on using in your RAIDZ array at the same time, because unlike other software RAID (or even hardware RAID), you won't be able to change it later. This inherently requires you to pre-plan your expansion. It is best to budget your hard drive money and save for the major sales and buy multiple shuckable external hard drives at once. This means you get maximum TB/$ spent and all the benefits of ZFS RAIDZ.
Note that the above limitations and pre-planning are only required if you want to have raid-like features. It is perfectly fine to use ZFS on a single drive and using snapshots + ZFS send/receive makes block-level perfect backups to a second drive (that can be different sized) while using minimal space, protect from ransomeware, and are faster than any rsync or a cp copy. Remember that ‘”RAID in any form is NOT a backup”’ - RAID provides uptime and availability in case of a drive failure. So when asking yourself "Should I use ZFS RAIDZ?” you really should be asking "Do I really need RAID” (Is uptime and availability in case of drive failure very important to me?) and "Can I budget/plan my purchases around buying multiple drives at once?”. If the answer to both of those questions is "Yes", then you can and should use ZFS RAIDZ. If no, but you still want uptime, use something else like Snapraid or mdadm. If no, but you still want all the features of ZFS and meet the requirements listed above, then use ZFS on single drives.
General Reccomendations
Not Recommended:
- Running ZFS on ancient hardware.
- Running ZFS on consoomer motherboards.
- Run ZFS without ECC Ram. If you can afford ZFS you can afford to get ECC Ram. No excuses.
- Run ZFS on underqualified hardware (shitty little NAS boxes, SBCs, etc).
- Use "Mutt" pools (Zpools with differently sized vdevs).
- Growing your Zpool by replacing disks. Backup your data elsewhere, create a new pool, and transfer the data to the new pool. Much faster. (You could theoretically use a USB drive dock provided your array is 5 disks or less).
DO NOT
- Run ZFS on top of Hardware RAID.
- Run ZFS on top of other soft RAID.
- Run ZFS in a VM without taking the proper precautions.
- Run ZFS with SMR drives.
DO
- Run ZFS if you have ECC ram and a sandy-bridge or newer processor
- Run ZFS if you care about having the best and most robust set of features in any file system
- Use snapshots (see syncoid, snapoid, and other handy tools to schedule and manage snapshots)
- Use ZFS send/receive for backups
- Sleep soundly with a smile on your face knowing that you have the most based filesystem in the world
Btrfs
Snapraid
Hardware RAID
If you bought an old used server with a RAID controller already installed, or perhaps you don't feel like messing with software RAID solutions, you have the option of using hardware RAID rather than software RAID.
Choosing a file system
XFS
ext4
NTFS
If you are using snapraid as your raid solution, using NTFS formatted drives is perfectly fine. With Snapraid you are usually pulling out random drives you have lying around, which are most likely to be NTFS formatted. Otherwise, we do not recommend using NTFS unless you are running a Windows server for some reason. It does not have the same level of support on Linux and UNIX based systems as ext4 and XFS.
unRAID does not support NTFS. If you are using unRAID you will need to use ext4 or XFS.
