Don't assume that just because the program is closed it is safe to remove it. Windows can take a bit to manage the data on those things, and if you pull it while it's working, you can easily corrupt it. Safely remove it all the time. You never know when windows will decide to go in randomly and index everything for the *censored* of it.