All USB devices can, by definition, be unplugged or plugged in without physical damage to either the device, port, or machine.
However, If the device is being used at the time, you'll have problems later. For example you shouldn't yank out a Thumb Drive while reading or writing files to it.
Personally I never use the "safely remove Hardware" option for Flash Drives, and I've not had problems, though of course past performance is no guarantee of future results, and many others have had problems.
Using the "Safely remove" option is ideal for storage devices, not because it requires a special removal step, or even benefits from it, but the fact that it will only succeed if nothing is currently using the device. Since if the device is in use when you unplug it you could lose data.
As for driver software, jason2074 covered that thoroughly.