There is a slight confusion with the terms you are using here. A "file" is a document or single item. A "folder" contains files and/or other folders.
The problem you have is that the Office dialogue boxes automatically scroll to the first real file in any folder. In your case, since you are viewing all files and you are configured to view system and hidden files, "desktop.ini" will be the first file in some of your folders.
I do not know a way of overriding this behaviour, but here's a quick tip: In the dialogue box, press Shift-Tab, to move the selection cursor to the file/folder list. Then press Home to go to the start of folder list. You can use the cursor keys to go up and down the list, Enter to descend into one of the folders, and Backspace to revert to the containing folder. All without touching your mouse.
Hope this helps.