Re: memory usage, Task manager doesn't show you everything. Because of the way memory is managed there are quite a few values that one would need to inspect. Also, some programs "preallocate" a bunch of memory for some reason, but they will only really slow down when more of it is used. (the "fail" of this is that they are trying to do the task that the memory allocator is supposed to be doing).
A good tool for this sort of analysis is Process Explorer. It can provide columns for various memory information on each process- Minimum and Maximum Working Set, Current Working Set, Shared Bytes, Private Bytes, and the number of GDI and USER handles... which actually brings me to another common cause: leaked handles.
With Windows drawing, things are usually drawn using Pens and Brushes. When a application uses a brush, it has to remember to delete it at some point; same with Pens, bitmaps, etc. If it doesn't clean it up, it leaks that handle. Pens and Brushes (and I believe Fonts) are GDI resources. In older versions of windows, there weren't very many handles to go around and you could easily tell when a program was misbehaving because everything would be affected (Windows might start drawing things in the default font, as one example).
The thing that makes restarting fix this is that With Windows NT, process termination will properly destroy all the leaked handles, too.
Leaked handles over time can cause slowdown because operations on other handles now cause GDI32 or User32 to sort through the leaked handles, because as far as they are concerned they weren't deleted and therefore they must still be being used for something.
Process Explorer can show GDI and user objects; I believe Task Manager can as well, so you can take a look at that next time and compare the numbers.
The Device thing for me has always been a common cause, though I've had Visual Studio become unresponsive after a few days of uptime that can only be fixed with a restart (though it is often using 1 or 2 GB of RAM in private bytes, too; probably something storing a lot of data (I blame Resharper).
Twilight is best pony.
A COMPROMISE!