We are still actively working on the spam issue.
Home server Original
Home servers are about learning and expanding your horizons. De-botnet your life. Learn something new. Serving applications to yourself, your family, and your frens feels good. Put your /g/ skills to good use for yourself and those close to you. Store their data with proper availability redundancy and backups and serve it back to them with a /comfy/ easy to use interface.
Most people get started with NAS. It’s nice to have a /comfy/ home for all your data. Streaming your movies/shows around the house and to friends. Know all about NAS? Learn virtualization. Spin up some VMs. Learn networking by setting up a pfSense box and configuring some VLANs. There's always more to learn and chances to grow. Think you’re god tier already? Setup openstack and report back to /hsg/.
Things that are online today might not be online forever. It's good to have a copy of something because you never know when it might get taken down due to copyright strikes.
Contents
- 1 Hardware
- 2 Networking
- 3 Software
- 4 RAID
- 5 Backups
- 6 Networking
- 7 Security
- 8 Uninterruptible Power Supply
- 9 Opening to the Wider Internet
- 10 VPNs
- 11 Monitoring
- 12 Miscellaneous
- 13 Showcase
- 14 External Links
- 15 See also
Hardware
What hardware you get depends mostly on your use case. A simple file server can be run on an SBC with a couple hard drives attached. If you want to do more fancy things like virtualization, streaming 4K movies, etc you are going to want better hardware. If you plan on using ZFS or Btrfs, server grade hardware and ECC RAM are recommended but not required.
Server options
There are many roads to the Home Server. Each one has upsides and downsides. It's up to you to decide what works best for your requirements.
SBC and NUC
For simple home server use, such as a file server or single user direct play PLEX server, these options might be an appealing and inexpensive and energy efficient option for your home server. Expect performance issues if you try to scale though, and don't expect to be able to run multiple virtual machines or do heavy transcoding, you don't have many options for expansion and little to no options for upgrading either. Forget a hardware RAID card or having any SATA ports at all. You likely won't be able to add much more ram, and definitely won't be able to increase processing power unless you go the clustering route and purchase multiple units. If your use case is compute intensive or would require expansion cards (like a GPU for example) SBCs are likely not a good option.
If you decide to go with an ARM-based board be aware that some software will not work because it is only available for x86-based CPUs. The reason is usually that the software is proprietary and was only ever released for x86. Among ARM-based SBCs the Raspberry Pi has by far the best software support as it has the largest userbase.
Connecting hard drives via USB docks may have some performance impact. Use USB 3 where possible and don't attach too many drives to a single port.
- Raspberry Pi
- Rpi4 recommended - Better Ethernet and more powerful than the 3b.
- Odroid
- Odroid N2+ recommended
- Ondroid HC4 storage server - Cheap two drive system
- Odroid HC2 is an option - if you don't mind dipping your toes into distributed systems.
- NanoPi
- NanoPi M4V2 More expensive than a Rpi4 but more powerful hardware with the option to install a x4 SATA HAT.
- Intel NUC
- Search your NUC here for more information on it
NUCs have significantly more power than a SBC, and are exclusively Intel-based. They run the gamut from small celerons to some of Intel's most powerful mobile chipsets. Very feature-rich, most all of them will include quicksync features for transcoding, and have some amount of expansion capabilities (adding/changing ram, additional SSD in some cases). NUCs will also be significantly more expensive than the above listed SBCs.
Repurpose Old Hardware
If you have an old gaming PC, workstation, laptop, or spare parts lying around, you might be able to get away with using them as your server (provided they are not too old). Performance and capability will vary wildly from machine to machine. When in doubt post specs in /hsg/ and ask.
Laptops are not really designed for 24/7 use, but their battery does act as a built in UPS to a certain extent. Preferably use one with USB3 or better, older laptops might only have USB2 ports which will bottleneck any attached HDDs.
If you already own the hardware, this option is free, which can be very appealing and a great way to get started. Keep in mind that a lot of old PCs are very power hungry (for example with Pentium 4 CPUs). In some countries this means that your power bill for this machine could be more expensive than the cost of some new cheap SBCs and their power bills combined.
Build Your Own
If you have the money, buying new hardware is a viable (but expensive) option. Knowing exactly what you are getting and peace of mind that you can RMA any DOA items, as well as reasonable shipping prices are good reasons to buy new. A combination of new hardware and repurposed older hardware is also an option if you are on a bit of a budget.
If you are looking to build a ZFS/FreeNas server be sure to get a motherboard and CPU that support ECC RAM. Server motherboards are recommended as they have many features such as IPMI, Intel NICs, NIC teaming support, and more.
Supermicro/Asrock Rack are good options. "Prosumer" boards are usually incredibly expensive and not worth the money.
Modern AMD Ryzen CPUs all "unofficially" support ECC RAM, just make sure your motherboard supports it. Pretty much all AMD CPUs also support virtualization.
If using Intel CPUs check ark.intel.com for information on your CPU. Some features to look for:
- Intel Quick Sync Video allows for hardware accelerated transcoding.
- VT-x and VT-d are must haves if you plan on using virtualization of any kind.
- ECC RAM Support
Buy Used
Buying Used enterprise hardware can be a cheap, but somewhat unreliable option. Waiting for a good deal might not be for everyone but the rewards are great. Tremendous amounts of storage potential at a relatively cheap price. Some rackmount servers will even come with drives preinstalled.
Be aware though, rackmount servers are usually pretty loud, and many older Xeons can be extremely energy inefficient. For most popular brands/configurations there are YouTube videos specifically for the sound. Search for "R720 noise" in YouTube and you will find videos of people putting their microphones up to them so you can assess the noise levels for yourself and your application. Don't let the noise deter you unless you plan to have this server in a living space in your home. Even if you do, anything 2U or 4U usually has the option for a "quiet mod" where you swap the fans with noctua's or similar, drastically reducing the noise the machines create.
Things to look out for when buying used:
- 32bit systems have a hard limit on Ram. Avoid at all costs.
- Some older legacy systems do not support UEFI and thus cannot boot UEFI OS's.
- Older hardware specs may become performance bottlenecks (Earlier SATA/PCIe/SAS/USB revisions).
- Some Very old (8-10 years) high end CPUs actually perform worse than modern low-to-mid end CPUs.
- Lack of support for potentially desired features such as QSV, 1GB/10GB Ethernet, m.2, etc.
- Some disk shelf/server backplanes are SAS only and won't accept standard Sata drives.
Good places to find old server hardware:
Case mods
Prebuilt NAS
Only buy a prebuilt NAS if you want to spend more and get less.
They are typically woefully underpowered for the price and you’re better served with a $65 Odroid than a $300 QNAP/Synology with a shitty Celeron and 1gb of ram. That said, they are the most noob friendly option with a GUI interface for setup.
Storage
Shucking
It's massively cheaper(sometimes) to buy WD Easystores or WD Elements (when they go on sale) than it is to buy an equivalent size NAS hard drive like WD Red/Iron wolf. Just remember: YOU VOID YOUR WARRANTY(if you live in freedom land, yuropoors retain their warranty as long as you still have the shell and can put it back together). If your drive fails you are most likely fucked. When you buy regular NAS drives you are basically paying more for the warranty.
Some other things to consider if you decide to shuck:
- Shucked drives under 8TB might be SMR drives.
- Shucked drives lack the middle mounting hole that most other drives have. You may need an adaptor for your HDD trays if your case doesn’t support them. Some cases might not have adaptors at all, research before buying your case!
- Some 8 and 10TB drives are air-filled rather than filled with helium. These air-filled drives can run significantly hotter than the helium ones. Check the model number with Crystal Disk Info before shucking, if it has an H it is most likely a helium drive. If you have airflow constraints in your case, it might be better to try and get Helium drives, otherwise it shouldn't be much of an issue.
3.3v pin issue
Shucked drives WILL NOT BOOT with most consumer power supplies. This is because of a feature on enterprise drives that lets administrators reboot hard drives by powering the 3.3 volt pin which isn’t used on consumer hard drives. Consumer PSUs, of course, always power this pin, so the hard drive will be stuck in an infinite boot loop and never power on. This can be solved by covering the first three pins on the hard drive with insulating Kapton tape.
- DO NOT use liquid electrical tape. This can damage the drive.
- DO NOT cut the SATA power cable this can damage the drive and your PSU.
- Molex to SATA adapters DO work but be careful, as some of the poorly made ones can catch fire. I wouldn’t risk it.
SMR v CMR
SMR stands for "shingled magnetic recording" It's an alternative method to conventional magnetic recording (CMR) that traditional hard drives use. While SMR technology allows for greater data density, they are also slow compared to CMR. These drives are bad for NAS use cases and especially bad for ZFS due to compatibility issues. Just avoid them all together. All Seagate NAS drives are CMR. Easystores/Elements 8tb and above should be safe.
SSD
SSDs are recommended for the OS and programs only, or for use in cache, or L2ARC cache/SLOG in ZFS.
Don't buy SSDs for main storage unless you want to spend tens of thousands. Check out the SSD buying guide for more on SSDs. If you do have a large SSD array, post the details in /hsg/ so we can all drool.
Expanding Your Storage
If you find you have run out of SATA ports on your motherboard but require more storage, there are a number of options for increasing the number of drives your server can support. The best and recommended approach is to use a SAS HBA with SAS to Sata breakout cables. Each SAS port can support up to 4 Sata drives (or even more if you use an expander). You can find used LSI SAS HBAs on eBay for relatively cheap which have ~2 internal ports, or 8 total SATA drives. Avoid SAS1 cards as they are far too old by now and have some limitations. If your case can no longer support more drives, you may want to look into buying an External SAS HBA, which will allow you to connect drives in an external enclosure directly to your server.
- Some videos on SAS controllers and cables which I found very helpful. If you are new to using SAS you should watch these:
Sata HBAs and port multipliers/Expanders are not recommended. They are garbage and not worth buying. SATA port multipliers specifically can cause issues when you try to us any kind of RAID with them.
There are some counterfeit LSI cards on the market, avoid Chinese sellers, sellers with no return policy, etc.
Drive Recommendations
Generally speaking, it's always best to buy the cheapest possible TB/$ drives you can buy while aiming to build redundancy and backups into your storage plan. This may be used drives(if you're willing to roll the dice on having no warranty), this may be shucked drives(If you're okay with shucking and taping pins), this maybe be new general purpose drive(Remember and stay away from SMR drive), and very rarely you might get a good deal on an "enterprise" or "NAS" rated drive for cheap, Just remember, NAS/enterprise drives come off the exact same production line as every other drive, with very small minor tweaks to firmware that is mostly insignificant for a typical homelab consooomer. Don't get conned into thinking you need a NAS drive for your NAS.
To sum it up, It's much better to have a 5% AFR(Annual Failure Rate)disks in a raid6/raidz2 array than it is to have a 0.1% AFR drive raid5/raidz1 array. If you can get the former cheaper, do it. Just remember, 3-2-1 backups and you'll never lose your data.
Check out these links for cheap drives - https://shucks.top/ & https://diskprices.com Also check out Backblazes drive failure data so you can compare some similar prices disks, spending a few $ more on a low AFR disk doesn't hurt, but don't go overboard.
Racks and Cases
- Home server case guide
- Lack rack - Meme-y but practical and cheap solution for rack-mount equipment. Be wary of putting too much weight on them though
Networking
Please share your networking setups and provide advice for other anons.
Routing guide (WIP): Home_server/Routing_for_retards
Ethernet
Ethernet cables come in different types, each designed for specific networking and data transfer needs.
Cat 5e supports up to 1 Gbps at 100 meters. This is the standard choice for most networks and building wiring. Higher categories such as 6, 6a, and 7 support faster speeds and reject electromagnetic interference better. Even if your LAN (your internal network) only supports 1 Gbps, using higher categories is beneficial. For more info, see the detailed Ethernet CAT Specs page.
The speed of your LAN is based on the network interfaces of your servers, the capabilities of your switches, and the speed of the communicating network cards.
Routers
ISP Provided: these types of routers can be very locked down, but most have the capability to enable "bridge mode". This is where all router functionality is disabled and it acts as a simple modem. You then plug your own router into it.
Consumer: Netgear, TP-Link, Linksys, Asus, etc. Base firmware is usually subpar but OpenWRT can turn them into a powerhouse.
Prosumer: Ubiquity, EnGenius, MikroTik. These should handle whatever you throw at them.
Enterprise: Usually overpowered for the homelab. All the caveats of of used servers apply.
Custom: Custom hardware (ie. a thin client or desktop) running OPNsense or PFsense. Very powerful machines capable of performing all network tasks. Steep learning curve but extremely useful in any home network.
Switches
Connecting your server to the internet requires a physical ethernet line. But what if you run out of jacks on your router? Get a switch.
There are two types of switches: managed and unmanaged. Managed switches can be configured and support advanced networking concepts while unmanaged switches are plug and play with no configuration capabilities. Unmanaged switches are cheaper than managed ones.
Don't cheap out on switches or else you'll be wondering why your network is much slower than it should be. Expect to pay at least $20.
Network Interface Cards (NICs)
Intel network cards are considered one of the best for servers due to their high performance, reliability, and advanced features. They are designed to handle heavy network traffic efficiently, making them ideal for server environments. Intel cards usually start at $30 used on Ebay.
Cheaper network cards tend to use Realtek chips in them. The key difference is that Realtek network cards offload much of the network processing to the CPU. This means that the CPU has to do more work, which can reduce the overall performance of the server under heavy load.
If you build a OPNsense/PFsense router, make sure you use Intel network cards. In addition to the performance concerns, BSD and the router software has compatibility issues with Realtek.
Software
Operating systems
There are many options for which OS to use for your server, Ultimately it depends on your needs and budget.
- Debian GNU/Linux supports software RAID and file systems like OpenZFS, and runs on pretty much anything.
- OMV is good enough if all you are storing is rarely accessed media and is GUI based if you prefer that over a command line.
- TrueNAS CORE (formerly FreeNAS) is BSD based and fairly simple to install and use but server grade hardware and ECC RAM is recommended.
- Proxmox and VMware ESXi if you want your server to be primarily about virtualization. If this is your only server, this may increase the difficulty in creating ZFS or raid pools. Not impossible, just more tricky.
See Home server/Choosing an Operating System for more information.
Linux
These are all server-specific or at least minimal operating systems without a desktop environment or other bloat preinstalled.
- Debian Stable is one of the best operating systems to use for a server. It is not too hard to manage, but at the same time customizable enough for your server's purposes. Has plenty of documentation.
- Ubuntu Server is based on Debian Testing. Slighty less stable than Debian, but has far less outdated software in its repos. Recent LTS releases have focused on providing heavy integration with Openstack. Does retarded things with packages and versions (lib*-ubuntu1.l2). Arguably the best option for users new and old.
- Template:Strike CentOS is kil. RIP
- Alpine Linux is an extremely lightweight hardened distro using musl and busybox instead of glibc and coreutils. Uses OpenRC instead of systemd. Commonly used as base for docker images thanks to its small size, but works well on bare metal too. Recommended, especially for more experienced users.
- Gentoo is usually too much trouble to be worth it, but it works and sees occasional server usage.
- Arch and other rolling release distros are not good choices as they are generally unstable and often break/change behavior on updates.
YunoHost
Debian-based. Pre-configured with a web interface (accessible through its local gateway) and an app catalog for server software. A great choice for beginners.
If you're lost, just go with YunoHost or Ubuntu. Use mdadm, ZFS on Linux (ZoL) or Snapraid for data redundancy.
Open Media Vault
Good for storing infrequently changed files like media files.
Like FreeNAS/TrueNAS, OMV is primarily a GUI tool, but it is Debian based and a command line is always a ssh away. All configuring of typical NAS thing is available to you in a GUI (samba/NFS/shares/user management/etc). OMV does not force ZFS on you however, like TrueNAS does. OMV is ideal if you want a GUI on a SBC or less powerful hardware.
Supports Snapraid as a plugin. Can be used with mergerFS to pool drives together.
- OMV Forums
- OMV Extras - Needed for mergerFS
- Installation guide
unRAID
Comes with it's own RAID solution that technically isn't real RAID because all parity is stored on one or two disks. Not free, you need to fork over some money to buy it.
Supports differently sized physical disks and adding hard drives to expand as needed.
Unraid 6.8.3-6.9.2
SHA256: 18F75CA34A39632DC07270510E453243753CFF302F3D5ADD4FA8813D4ADB304D
magnet:?xt=urn:btih:180782e4ff3e00b7efc8a0529239b896e0557f72&dn=unraid692.7z
BSD
All are highly regarded by their users.
TrueNAS CORE
TrueNAS CORE is the free version of the premium TrueNAS and successor/replacement for FreeNAS. TNC is a FreeBSD based OS that utilizes ZFS for storage and has many available plugins for things like PLEX, bitorrent, and more. Has simple, easy to use GUIs to set up your services such as samba shares, etc.
Keep in mind it will install to the ENTIRE DRIVE and you won't be able to use the install drive for anything else. A small, cheap, M.2 SSD is a good option for the OS drive. Server grade hardware and ECC RAM is recommended.
Hypervisor
You may want to consider using an OS designed for virtualization/containerization. Virtualization allows you to run multiple independent operating systems on the same hardware simultaneously. You can use this for home lab, or game servers, or even virtualize your desktop instead of using a big tower.
Containers add the ability to isolate processes to make a more stable server, and also allow you to migrate services from one server to another on the fly.
Proxmox
A GNU/Linux based Virtualization Environment that has built in ZFS support. Utilizes KVM, QEMU for virtual machines and LXC for containers.
Also Supports Ceph and GlusterFS for distributed storage and clustering.
Good alternative to VMware, but is lacking in some areas. Good enough for most people's needs.
ECC RAM is recommended as per usual with ZFS.
VMware ESXi
If you've ever worked in a datacenter on managed IT for big business you will be familiar with VMware ESXi, it's the most popular, feature rich hypervisor available. Unfortunately, it is not free, and only has a limited free tier with 8 core per VM limit. No vSphere, or most vStorage options like vMotion and distributed switching. For most people this is okay, but if you are a home-server enthusiast you might want to play around with all the features they have available. It's an excellent option unless you don't like to use proprietary software or don't want to go through the trouble cracking to get all of the features on the latest version.
If you use version 6.5 or 6.7 you can use this key to unlock all these features.
- vCenter: 0A0FF-403EN-RZ848-ZH3QH-2A73P
- vSphere: JV425-4h100-vzhh8-q23np-3a9pp
VMware 7.0 has dropped support form westmere-EP/gulftown (x5xxx) CPU's. If your system has these old CPU's you should consider upgrading to something later than Sandybridge if you want to use the latest version of ESXi.
SmartOS
SmartOS is not Linux, nor is it Solaris (but it is Illumos underneath). It's a type 1 hypervisor platform that is/was the core of Joyent's public cloud platform (has since been sold off to MNX who are claiming continued support for opensource involvement).
Similar to TrueNAS, ZFS is not an opt in feature, and unlike most operating systems it does not require a installation disk, the system is entirely ephemeral running from a USB stick which can be pulled out at anytime. All of your VMs and other persistent data is kept on whatever zpool you name "zones", you are free to add and remove other pools and have all of the ZFS features available in the gz ("global zone") which is the base environment you are given to manage the system. What you can't do in the gz is install most packages, setup additional users, make any persistent changes to config files or run services.
Instead everything you do happens under zones, these are very similar the BSD jails or docker containers, the main difference is they are a first class kernel feature and have exceptional security and efficiency properties. As they are running bare metal with sandboxed zfs datasets it's possible to host samba and NFS shares of the same filesystem the VMs are stored on, in fact there's no reason you couldn't run multiple samba zones with access to separate areas of storage, this means even with a rootkit installed on one of your samba servers the other would remain completely isolated.
If you don't care for zones you can also setup full on HVM instances using either KVM or Bhyve, the later being far more performant and able to run the most recent Windows versions. VNC video consoles and serial ports are automatically setup whenever you boot a VM for remote management.
The gz comes with 3 cli tools for doing day to day tasks: vmadm, imgadm and zlogin. Running man followed by one of these commands will get you some very decent documentation, so l3rn to read.
Suggested reading:
- Bryan Cantrill sperging out about containers
- How to delegate datasets for use as a file server and Creating said file server zone
- Contact information everyone on IRC is very helpful so don't be a cunt
SBC Operating Systems
If you are using an SBC or NUC for your server, these are potential options to use over standard (GNU/)Linux distros.
DietPi
Extremely lightweight, extremely optimized version of Debian, and best of all, extremely easy. Can be installed on any SBC. Comes with a software centre with automatic configuration and optimization for your hardware, maximizing performance. CLI-based, but very retard-friendly and includes some nice menu systems.
FreedomBox
Runs on virtually any SBC.
Setup is incredibly simple. Installing new software and services can be done with the click of a button.
Lets you easily share files, host websites, sync files, and more. The number of available applications are a bit limited however.
YunoHost
Debian-based. Like FreedomBox, incredibly simple.
Containers
Containers are a method of isolating running software from both the host OS and other software. You may also hear them called Jails or Chroot Jails if you are running some variant of BSD (such as TrueNAS CORE/FreeNAS).
There are a number of reasons why you would want this:
- Less overhead than standard virtual machines because you aren't virtualizing the kernel.
- Isolated software cannot interfere with each other or the host. If a container crashes it won't effect anything else.
- Like VMs, containers are portable. You can create a container, configure it however you want, and deploy it anywhere.
- Like VMs, removing containers and starting from scratch or a backup in the event of a fatal crash is easy.
- Docker and Podman containers are incredibly easy to deploy and you can find pre-configured container images online.
Best practice is to keep the base OS as clean as possible and install each individual application (such as PLEX, Samba, etc) in their own container. This makes your server much more stable since there is virtually no chance of a containerized application crashing your server, or an installation gone wrong from ruining your host OS.
LXC and LXD
LXC is the standard Linux containers. Available on most distros. You will likely be using these if you are running a Proxmox server. Since Linux containers are essentially just semi-separate instances of Linux, you can't run Windows programs in them without using WINE.
LXD is a newer, more user friendly version of LXC. Has better management options for containers.
Docker
Instead of running as though it was an entire OS like LXC, Docker only virtualizes a single application. Can run on Windows as well as Linux. You will still need WINE to run Windows program on Linux. Docker is free software. Docker 'desktop' is proprietary software, which is available free of charge for "personal" usage, but requires licensing for many features and commercial usage.
Podman
An alternative to Docker. Those using Docker can easily switch without issues. Unlike Docker, it does not use a single large server daemon. Uses "pods" which can contain more than one container.
Jails
Jails are BSD's version of containers. Since TrueNAS CORE/FreeNAS is FreeBSD based you will be using these instead of LXC/LXD.
- TrueNas Jail documentation
- Give Jails access to host storage - Jail version of Bind mounting
- FreeBSD Jail documentation
Server software
For a greater range of self hosting solutions and services see awesome self hosted software.
System administration software
For a greater range of sysadmin solutions and services see awesome sysadmin software.
Media Streamers
The most common applications in the Media Server landscape is Plex, Jellyfin, Kodi and Emby.
If you want a feature comparison between these to find out which is right for you, check out THIS handy table
Want to transcode multiple streams simultaneously? Check out these hardware requirements
Have an Intel iGPU and want to offload transcoding? Check out this resource
RAID
RAID (Redundant Array of Independent Disks) is a technique where multiple physical hard drives are combined into a single logical unit for the purposes of redundancy, speed, or both. Data is stored in different places on multiple hard disks to prevent loss in the case of a drive failure.
RAID is an expansive subject so it's gotten it's own page.
ZFS
Everyone loves ZFS. Want to know more?
Backups
For more info on generic backups, see the Backups page.
This dongle is a Proxmox Backup Server
https://www.proxmox.com/en/proxmox-backup-server/overview
Proxmox developed a special server OS designed for one thing: storing backups. It has all sorts of very handy features that makes the backup and restore process effortless. It can be tightly integrated into the Proxmox Virtual Environment to enable one-click backup and restore. Proxmox also provides standalone Debian clients that bare-metal hosts can use to back up to the server.
One of the main advantages is its ability to perform incremental backups, which means it only backs up changes made since the last backup, significantly reducing the time and storage space required. The backup server can also automatically test backups to ensure they will restore successfully if needed.
Using the Proxmox Backup Server eliminates the hassle of backups. Even if you only have one server, running a Proxmox Backup Server is a worthwhile investment.
Networking
You will see numbers like 10/100/1000 on networking hardware. This refers to the data transfer speed of network hardware and is measured in megabits per second (Mbps). Make sure your hardware supports 1000 Mbps (1 Gbps). If you only see 10/100 the hardware only supports a maximum of 100 Mbps. Trash it.
Ethernet
Ethernet cables come in different types, each designed for specific networking and data transfer needs.
Cat 5e supports up to 1 Gbps at 100 meters. This is the standard choice for most networks and building wiring. Higher categories such as 6, 6a, and 7 support faster speeds and reject electromagnetic interference better. Even if your LAN (your internal network) only supports 1 Gbps, using higher categories is beneficial. For more info, see the detailed Ethernet CAT Specs page.
The speed of your LAN is based on the network interfaces of your servers, the capabilities of your switches, and the speed of the communicating network cards.
Switches
Connecting your server to the internet requires a physical ethernet line. But what if you run out of jacks on your router? Get a switch.
There are two types of switches: managed and unmanaged. Managed switches can be configured and support advanced networking concepts while unmanaged switches are plug and play with no configuration capabilities. Unmanaged switches are cheaper than managed ones.
Don't cheap out on switches or else you'll be wondering why your network is much slower than it should be. Expect to pay at least $20.
Routers
ISP Provided: these types of routers can be very locked down, but most have the capability to enable "bridge mode". This is where all router functionality is disabled and it acts as a simple modem. You then plug your own router into it.
Consumer: Netgear, TP-Link, Linksys, Asus, etc. Base firmware is usually subpar but OpenWRT can turn them into a powerhouse.
Prosumer: Ubiquity, EnGenius, MikroTik. These should handle whatever you throw at them.
Enterprise: Usually overpowered for the homelab. All the caveats of of used servers apply.
Custom: Custom hardware (ie. a thin client or desktop) running OPNsense or PFsense. Very powerful machines capable of performing all network tasks. Steep learning curve but extremely useful in any home network.
Network Interface Cards (NICs)
Intel network cards are considered one of the best for servers due to their high performance, reliability, and advanced features. They are designed to handle heavy network traffic efficiently, making them ideal for server environments. Intel cards usually start at $30 used on Ebay.
Cheaper network cards tend to use Realtek chips in them. The key difference is that Realtek network cards offload much of the network processing to the CPU. This means that the CPU has to do more work, which can reduce the overall performance of the server under heavy load.
If you build a OPNsense/PFsense router, make sure you use Intel network cards. In addition to the performance concerns, BSD and the router software has compatibility issues with Realtek.
Security
Unlike a desktop, a server is always working, accepts connections from the internet (your desktop is normally firewalled and doesn't have any ports open) and is easy to discover (especially if you send mail from it). It's under a bit more risk, and its worth thinking about what intrusions you will try to prevent and how. Refer to Security#Threat_analysis to understand how and what threats you can mitigate.
Basic measures include:
- Privilege separation
- If you are behind a router, only forward ports you need
- Your firewall should reject all traffic which isn't either in response to an existing connection, or destined for a forwarded port
- Make sure to keep your router firmware updated, as vulnerabilities are often patched in newer versions (at least, from the companies which bother even releasing them). If your device doesn't receive support in the form of firmware updates and security fixes, consider running community-maintained firmware such as OpenWRT
- Regularly update software and kernels when they become available for your distro (it is far better to fix what updates break then get owned)
Uninterruptible Power Supply
An Uninterruptible Power Supply (UPS) is a device that provides emergency power when the main power. It ensures the continuous operation of connected devices and prevents data loss or damage that can be caused by sudden power outages. A UPS protects against outages a few seconds in duration and can give the servers adequate time to shutdown in the event of an extended outage. A UPS also provides power conditioning, protecting against power surges, voltage spikes, and frequency variations, which can harm the connected devices.
A basic UPS can be bought for $60 and powers a couple machines. A full-featured UPS is around $200 and includes advanced monitoring and signaling to connected machines via NUT (you should set up the auto-shutdown feature if your UPS supports it).
It's highly recommended you purchase a UPS to protect your machines, even if you don't expect any outages.
A UPS's capacity is measured in Volt-Amperes (VA). A capacity of 400 VA is good for one or two desktops. A capacity of 1500 VA can run multiple large servers.
Popular brands are CyberPower and APC. Always by new UPSes, never used.
Opening to the Wider Internet
Just like how your home address is visible to anyone who walks by, your IP address is visible to anyone who knows how to look for it. This includes IP scanners, which are tools that can scan the internet. Don't expect anything you host to be secret if you let it out of your firewall.
This is fine for things like personal websites, but hypervisor dashboards and other admin panels should never be exposed.
Some people think that if you host something on a very high port number (like the port 55432), you'll be safe. That's not true and is a false sense of security. Security though obscurity is never safe.
The SSH port (port 22) is a very popular scanning target. If you allow SSH out of your firewall then you should disable password authentication and only use certificate-based authentication. Using fail2ban to block repeated authentication failures is also a good idea. Don't try changing the SSH port, that's a false sense of security. You should avoid exposing your SSH server and only have it accessible on your local network which you will use a VPN to connect to.
Cloudflare and Hiding Your IP
DNS records can expose your home IP address because when you register a domain name, you need to provide DNS records that map your domain name to your IP address. This is how the internet knows where to send traffic for your domain.
This is a problem because your IP address can reveal information about your location and your internet service provider. It can even be used to launch targeted attacks against your network.
To hide your home IP, you need to proxy your traffic from a middleman. Most of the internet runs on Cloudflare, who provide DNS and DDOS protection for free.
Dynamic IPs
A dynamic IP address is an IP address that changes from time to time, unlike a static IP address. Most home networks are likely to have a dynamic IP address. You can use ddclient to update your DNS records to the correct IP when it changes.
ddclient supports Cloudflare: https://www.davidschlachter.com/misc/cloudflare-ddclient
CGNAT
Carrier-Grade Network Address Translation (CGNAT) is a technology used by Internet Service Providers (ISP) to share a single IP address among multiple customers. It is a response to the shortage of public IPv4 addresses.
When it comes to hosting a server on your home network, CGNAT can cause problems. If you're behind a CGNAT, you don't have a unique public IP address that the wider internet can use to reach your server.
You can use Cloudflare Tunnel to proxy your traffic through the CGNAT setup and directly to Cloudflare. The DIY way is to run a cloud VPS and use a VPN to connect back to your home network.
Reverse Proxy
Nginx is a powerful web server and reverse proxy server. When used as a reverse proxy, it can handle requests on behalf of backend servers.
A reverse proxy allows you to serve multiple applications on different servers from a single forwarded port.
For each internal service, you would host them on a separate subdomain (like plex.example.com, nextcloud.example.com).
More info at Home_Server/Reverse_proxy
VPNs
As you probably already know, a Virtual Private Network (VPN) is a service that allows you to connect to the internet in a secure way by providing an encrypted connection.
This technology can be used to remote into your home network and access your servers. By connecting to a VPN server that is set up on your home network, you can access your home servers as if you were physically present in your home. This is neat because it means you don't have to expose your internal services (hypervisor dashboards, admin panels, SSH, private services only you use) to the outside world. You only have to port forward your VPN port and you can access your home network from anywhere.
Make sure you check out overlay networks, they're really cool.
WireGuard
WireGuard is an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec, while avoiding the massive headache. It intends to be considerably more performant than OpenVPN.—WireGuard, WireGuard homepage
WireGuard also has Linux kernel modules, making it a very performant VPN. It's also designed to not respond if a peer fails to connect which keeps IP scanners from detecting what is running on that port.
OpenVPN
An older, classic, VPN server. Stick with WireGuard unless you need OpenVPN specifically.
Overlay Networks and Mesh VPNs
An overlay network is a network that is built on top of another network (such as the internet) and sends traffic through that infrastructure. The key benefit of overlay networks is that they can be deployed without changing the underlying physical network. Instead, they can be implemented through software in the nodes of the network. No port forwarding is required.
Mesh VPNs use a peer-to-peer model to create a secure shared environment for their users. They consist of nodes that send traffic between themselves rather than through a central server.
How do these two technologies connect you to your home servers? It eliminates the need for a central server and allows your servers to communicate across any network boundary and without opening any ports. Traffic is encrypted between nodes which is a convenient layer of extra security.
Despite their distributed nature, overlay networks and mesh VPNs always require some sort of cloud server with a static IP to help nodes cross NAT boundaries and discover each other. Running multiple of these cloud discovery servers reduces the single point of failure.
Warning: If the addressing of your overlay network conflicts with the addressing of the physical network you are connected to, traffic won't be able to flow. Make sure you pick an IP range for your overlay network that won't conflict with other networks. 100.64.0.0/10 is a good choice and recommended by Tailscale.
The two leading overlay networks are Nebula and Tailscale. Both are excellent choices.
Nebula
Nebula is an open source overlay network developed by Slack to connect their many datacenters. It has a "focus on performance, simplicity and security". Simplicity is Nebula's main selling point and it is extremely easy to get a Nebula network up and running. Just generate certs and set up a "lighthouse" (what Nebula calls the discovery cloud server).
A lighthouse requires very few compute resources and you can easily run a very large network off of the cheapest cloud VPS you can find (they suggest the $5/mo DigitalOcean server).
Nebula also features a per-node stateful firewall that enables network segmentation, something other overlay networks lack.
Nebula does not support IPv6 inside the overlay network. The GitHub issue tracking progress on implementing IPv6 in Nebula.
Tailscale
Tailscale is built on Wireguard and similar to Nebula in its capabilties.
The biggest difference is that Tailscale has strong user-based SSO authentication. This introduces additional complexity that may not be necessary for the home environment.
Comparison between Tailscale and Nebula.
Tailscale's discovery server is called a "coordination server" and is not open source. Headscale is an open source drop-in replacement.
Tailscale supports IPv6 inside the overlay network, greatly reducing the likelihood of an address conflict with the physical network.
Doing Cool Things With Overlay Networks
Tired of trying to remember what IP address is what? Run an internal DNS server only accessible through your overlay network and assign your hosts easy to remember domain names! Use .nb if you are on Nebula or maybe .ts if you use Tailscale. Then run Nginx Proxy Manager on your lighthouse or coordination server and enjoy free, unique domains only you can access!
Monitoring
Monitoring allows you to detect and send alerts when things break rather than having months go by before discovering something is wrong. Even if you have only one machine a monitoring platform can be helpful. Some good options include Icinga2, Zabbix, and Uptime Kuma.
If you have something that can stop working, add a monitoring check to make sure it keeps working. Eventually, your entire infrastructure will be monitored and you can have the confidence that things really are working correctly when your monitoring dashboard shows green.
Check scripts in Icinga2 and Zabbix are used to perform specific checks. These scripts can be written in any programming language and return an exit status code and a status message. There are many excellent scripts on GitHub and it's super easy to write your own.
Miscellaneous
Showcase
External Links
- Home server hardware - Hayden James' home lab setup
- STH Forums - Good general resource for server questions
- Learn Command line
- HP T620 plus - Decent cheap computer. You can use it as a VPN, pfSense firewall, and more:
- Tiny Certificate Authority For Your Homelab
