This can happen if you have antivirus software scanning incoming mail. I've seen it with Norton's email scanner most often, but a couple of times with other AV packages.
Unless it is ALL email ALL the time then I would say that's the most likely culprit.