Great news - I noticed what was wrong
In the "Regional and Language Options" under the Advanced tab it asks for the "Language for non-Unicode programs"
I assumed it meant it wanted the default language, which for me is English - United Kingdom.
What it really means is that for non-Unicode programs such as my Chinese applications, what character encoding should be used to display menus and dialogs which aren't in English.
As soon as I set this to Chinese PRC, rebooted, and re-installed the applications, everything now displays correctly.
That is for Unicode English it displays in English, and non-Unicode Chinese displays in Chinese.
Yippy