Xvfb is a virtual frame buffer for X windows, as I'm sure you know.
One uses it on a headless workstation so that an X application can draw into a piece of memory and not whine that there is no display on a given workstation.
I know of no security problems with Xvfb myself -- remember, it's nothing more than a piece of virtual memory pretending to be a video display card. There's not a lot to go wrong.
Now having said that, X has its own set of security issues. A decade ago, these were a much bigger deal than they are now. Standard installations of X lock down things pretty tightly. But those are things you'd have to worry about no matter what.
A search of Bugtraq for xvfb turns up two references to X11 cookie hijack problems (dating from 1998) and an XFree86 3.1.2 problem from 1996. That's a rather clean bill of health.
This was first published in April 2002