the background isn't always black, you can select any background out of the 16 colours. Black is the default.
Come to think of it, your dislike for VGA is probably not from any real issue other then simply not liking to program it.
Additionally, CGA only had TWO, not 4, official palettes.
1. Magenta, cyan, white and background color
2. Red, green, brown/yellow and background color