I don't think a light something like Xfce will take that much and you don't have to boot into it.
Just install it and if you want to use it (for example, to use a program that requires X, then just $> startx
It would be a nice-to-have, but I'm not a fan of gdm, etc. where they force a GUI these days.
Heck, you can't even stop it through init.d or service any more. If I want to replace video drivers or something and modprobe -r the old one, it makes it quite difficult. There *was* some non-intuitive way to do it in Debian.
Also, what in world happened to X's config? It used to make sense... >.>