Reason #1: To save space
The splitting of the distros like this is really more to do with how much cruft you have to download and install on the system. There's really no difference between the server version and the desktop versions except for which software packages are bundled on the media (DVD) + and what is installed by default when you use the particular versions.
As an example you could easily install CentOS using the server or minimal editions and then once the system is up and functioning, run a command such as:
  $ yum groupinstall '..Desktop Environment..'
to install all the packages required to make it a fully functioning desktop edition.
Reason #2: To specialize the distro installation
As to what else is missing, well that really depends on each distro. With server distros, they're usually trying to cut out software to make the download smaller and more manageable. So things like a web browser, LibreOffice, and anything GUI related will typically not be included in a server edition. They'll also omit non-GUI software like compilers, such as gcc, and a whole host of other software, which really have no business being included in a truly server install.
Some Examples
On the various distros, such as with anything Red Hat based like CentOS or Fedora, you can use the yum command to get a list of the various groups of software:
$ yum grouplist
Available environment groups:
   Fedora Server
   Fedora Workstation
   Fedora Cloud Server
   KDE Plasma Workspaces
   Xfce Desktop
   LXDE Desktop
   Cinnamon Desktop
   LXQt Desktop
   MATE Desktop
   Sugar Desktop Environment
   Development and Creative Workstation
   Web Server
   Infrastructure Server
   Basic Desktop
   Minimal Install
Installed groups:
   C Development Tools and Libraries
Available Groups:
   3D Printing
   Administration Tools
   Audio Production
   Authoring and Publishing
   Books and Guides
   Cloud Infrastructure
   Cloud Management Tools
   Cloud Server Tools
   Container Management
   D Development Tools and Libraries
   Design Suite
   Development Tools
   Domain Membership
   Editors
   Educational Software
   Electronic Lab
   Engineering and Scientific
   Fedora Eclipse
   FreeIPA Server
   Games and Entertainment
   Headless Management
   LibreOffice
   MATE Applications
   MATE COMPIZ
   Medical Applications
   Milkymist
   Network Servers
   Office/Productivity
   RPM Development Tools
   Robotics
   Security Lab
   Sound and Video
   System Tools
   Text-based Internet
   Window Managers
Done
and then see what's included in each using this command:
$ yum groupinfo "X"
-or- in later versions of Fedora/CentOS, for example:
$ yum group info "DNS Name Server" 
Group: DNS Name Server
 Group-Id: dns-server
 Description: This package group allows you to run a DNS name server (BIND) on the system.
 Default Packages:
   +bind-chroot
 Optional Packages:
   bind
   dnsperf
   ldns
   nsd
   pdns
   pdns-recursor
   rbldnsd
   unbound
You can also get a summary of what groups you have installed using this command:
$ yum groups
Available environment groups: 15
Installed groups: 1
Available Groups: 35
Done