Couldn't find much info on the net concerning the NTVDM problem in Windows XP. NTVDM is the 16-bit virtual machine inside Windows needed to run 16-bit programs. If you have legacy programs, it might be called into action, but many people will never even see it.
Try seeing if any entries in the event viewer give any hints as to what is going on:
Control Panel==>Administrative Tools==>Event ViewerCheck both the Application and the System logs. Maybe, just maybe they'll offer a hint of the problem. I'm quite sure the SP6 installer is a 32-bit program.
This problem only gets stranger and stranger.