I was able to remove the desktop environment include with Raspbian by first removing x11-common and then removing my ‘stale’ packages.

sudo apt-get remove --purge x11-common
sudo apt-get autoremove