X sounds like Z. When you say X-Xavier you are saying the letter of the alphabet versus sounding it out phonically. Just like K and C have similar sounds. If there was a hyphen after the X then it becomes EX. For example: x-ray
No one says:
ex- Xerox for Xerox
ex- xylophone for xylophone
ex-xanadu for xanadu
Please see the dictionary to see and hear the proper pronunciation of the name Xavier. You can also hear the proper pronunciation on dictionary.com.