Do you remember which order you installed your server and client?
Also I would double check the file type options under the windows explorer folder options.
Server first, then FMP.
The files display the correct icon and since the garbled installer dialog is asking for an installer file for Filemaker Pro 10, I would assume the file type settings are not the issue.
Problem is solved and I think I have a bug report now for FMP folks to consider.
I had downloaded the updater to update FMP 10 v1 to FMP 10 v2. I ran the updater but couldn't complete the process as it requires the install CD and My boss had put that disk away where I couldn't get to it. I then cancelled the install process. Apparently that cancellation didn't return all system settings (probably in the registry) to their original values. Once we got the Installer CD in place and ran the updater, this problem disappeared.
Thank you for documenting this problem. I have forwarded this entire thread to our Development and Software Quality Assurance (Testing) departments for review.