This type of thing requires all sorts of convolutions to get working in an actual programming language- such as C/C++, involving first determining wether a previous instance exists, getting it's processID, grabbing the top level window of that process, and sending an Application message to it with the command line arguments of the new instance, and then the new instance quits. the old instance processes the message and loads the command line- for example file names. Some would say to just use KillProcess on the old instance. Well that would A: destroy everything changed in the original documents in the first process, and B: possibly leave open handles to files and devices.
Java has it easy since the VM will do all the real work.