If you are running a program in XP compatibility mode, it must be 32-bit and therefore would never be able to access more then 2GB of RAM simultaneously. (Actually, 4GB, but that's 4GB has the upper 2GB reserved by the system)) However, memory beyond the ~3.5GB "limit" will still be allocated by the memory manager. Remember, a "pointer" or memory handle is only valid in a single address space; an address/pointer in one program can point to a piece of data while the same pointer number in another program could point to something completely different; each 4GB address space (32-bit programs) is completely independent. 64-bit programs have a much larger address space, but in both cases, the actual memory address and therefore where it is stored in physical memory is not actually something one can discover, and would be rather silly to try to do anyway because unless you lock a piece of memory the memory manager could swap it out at any time to virtual memory and swap in something else for some other program, meaning that that segment of physical memory is now in use by some other program.
The only thing "compatibility mode" does is configure a number of App-Shims that are commonly needed by programs written for XP.