The problem with browsers today is their cache policies seem to be trash.
A browser cache should not simply bloat until you have to close the browser. It should only keep information in memory that is actually useful. When you have 3GBs of cache information for your web browser being stored in memory for "performance" on a system with 4GB of RAM you need to reconsider your assumptions about performance.
EDIT: by way of example: I have 3 tabs open in Firefox.
It is using 2 Gigabytes of Memory. Why? Because it's storing a crapload of images, text, stylesheets, etc. from pages I've visited this session.