It sounds like FileMaker is ignoring the OS "default" application. One question is whether either or both of the applications is already running, though that should not matter.
If this is on a Mac, you could use AppleScript to tell the preferred application to open the file, if the application supports this most basic AppleScript command (Open Office may not). Worth a try though.
You would need to tell use whether the doc file has been Inserted as "embedded" or "reference only" into the container field.
Another tack would be to reinstall Open Office. In general (sometimes) the last application installed will become the "default" in cases of overlap. On Macs, the "launchd" system is somewhat complex, a moving target (changed in 10.6), and a little hard to control; especially when some applications (from large companies not named) decide to be a little heavy handed when they're installed.
There is a Preference Pane you can install to control this, RCDefaultApp-2.1.X. But I haven't had to use it for a while. Usually telling the OS to "Always open all files like this" with "Open Office" will fix it. The option is in the Get Info for a file.
Non of the apps where running while I tested it.
I'm using Microsoft, and embedded the document. Reïnstalling Open Office doesn't seem to help.
When I try adding it as a object, I have the problem. Adding it as a file solves the problem because you are forced to download the file first, and then upload it again, but there should be a better way to solve this problem...
Another option is to insert the file with the "store by reference" option. Double clicking the container field then opens the file--but you'll need to check and see which app opens it.