I've have always defragged on a regular basis and have never seen a noticeable change in performance on any of my machines but that's probably because I do it on a regular basis.
However, a friend asked me to look at his computer because it had a hard time playing a certain game. The game would start and stop constantly. It was almost unplayable.
He had the computer for 3 or 4 years. I asked him when the last time he defragged. He said what's defrag?
I did a diskclean and defragged twice for good measure, and then the game ran fine.
It depends on the circumstances, but defrag can make a difference.