It could be possible that your internet connection is not fast enough. I have DSL, and when my neighbor next door needs help, we use Remote Desktop. It's slow for me. This is becuase you're using the internet to take information from one computer to another. That's a lot of information. If the server seems to be making things slower, propose this to IT, and see what they say.
Flame