Home server/Choosing an Operating System

The most popular choice here is by far any flavour of Linux. BSD is sometimes considered better. Using MacOS will earn you derision from Linux and Windows sysadmins alike. Don't use Windows.

You may also consider using an OS designed for virtualisation/containerisation - see #Hypervisor.

Linux-based

These are all server-specific or at least minimal operating systems without a desktop environment or other bloat preinstalled.

• Debian Stable is one of the best operating systems to use for a server. It is not too hard to manage, but at the same time customizable enough for your server's purposes.
• Ubuntu Server is based on Debian Testing. Slighty less stable than Debian, but has far less outdated software in its repos. Recent LTS releases have focused on providing heavy integration with Openstack. Does retarded things with packages and versions (lib*-ubuntu1.l2). Arguably the best option for users new and old.
• Scientific Linux
• CentOS is basically a free community supported version of RHEL. Very stable, and very popular thanks to its links to Red Hat.
• Alpine Linux is an extremely lightweight hardened distro using musl and busybox instead of glibc and coreutils. Uses OpenRC instead of SystemD. Commonly used as base for docker images thanks to its small size, but works well on bare metal too. Recommended, especially for more experienced users.
• OpenSUSE
• Slackware
• Gentoo is usually too much trouble to be worth it, but it works and sees occasional server usage.
• Arch and other rolling release distros are not good choices as they are generally unstable and often break/change behaviour on updates.

If you're lost, just go with Debian or Ubuntu.

BSD

FreeBSD is the most popular, OpenBSD has a bigger focus on security, NetBSD is another option. All are highly regarded by their users.

Hypervisor

• SmartOS is what you get if you take Opensolaris, KVM and Linux syscalls and expertly stick them together. If you don't know if you need this, you probably don't.
• RancherOS the memeiest around. Has everything you need to run docker, a web UI and nothing else. Web UI for administration is quite good. Overall good, but Alpine+Docker does essentially the same thing (and the web UI is available seperately). Recommended for memes.
• CoreOS very good shit for advanced dockerfags

Additional

You could also consider a NAS-centric operating system for a home server. FreeBSD-based FreeNAS or NAS4Free are common choices. Both are free software designed for the lowest common denominator, with simple GUIs to set up your services.