Yes it is going to be a problem. The fat on your fingers has now formed a layer between the CPU and the thermal grease, prohibiting heat transfer as intended by the manufacturer. It might also become hot and leave ugly marks.
I'd replace it but if you think thermal levels are acceptable after 48 hours, you could leave it on I suppose ...