Defragmentation is something that many people think is a magic speedup spell, but it sounds like you are wise enough to doubt this. If a disk is very badly fragmented, then it can slow a system down quite a lot, and defragging can bring it back to normal (as long as the disk is not too full) but if the state of fragmentation is not too bad, then as you have noticed, not much difference (if any) will be seen. Also you don't need to do it very often.