So, I was able to replicate this twice in in a week.
The Nvidia card that I use has a fan on top of the heatsink. If I turn off (shutdown) the Core and restart it, the fan doesn't start up because it is just crappy and needs a nudge. And if the fan is not running, Xorg is running at 100% CPU utilization. If I blow into the fan and it starts to spin and starts cooling the adapter, the processor is back to normal. I am wondering if this is just purely heat related or if there is a fan speed sensor on the video card and it will turn off the GPU if the fan is not spinning. Odd, but just an FYI for others that are having high CPU utilization with Xorg.