USB devices are powered through USB port, and also the data is transferred through it.
In theory, by simply yanking USB device out of the port, the damage may occur to the port, and to the data on USB device.
However, the chances are really slim.
In older Windows versions, and in early XP version, there was an issue of delayed writing (write caching) to USB devices, so the system was less busy.
If that transfer was interrupted by removing the device, the data had a good chance to get corrupted.
In updated XP, or Vista, write caching is disabled by default, so, if you're not using any program, which is actually using USB device at the moment, removing USB device without using "Safely remove", shouldn't cause any problems.
Personally, I never use it, but, if you want to be double safe....
