2017 Edit: There are more options now as seen at Who's got the other end of this unix socketpair?. With Linux 3.3 or above and with lsof 4.89 or above, you can replace points 3 to 5 above with:
lsof +E -a -p 1237 -d 31
to find out who's at the other end of the socket on fd 31 of the X-server process with ID 1237.