To avoid the "file clutter", use export field contents to export the file to the temporary folder. Copies exported to that location are not retained.
The double click method only works if:
a) The file was inserted with Insert File and with "store a reference" specified
b) the container field is NOT optimized for interactive content
c) external storage is not specified for the container field
Thus, it's probably simpler to use the export field contents option, but specify your temporary folder as the location to which to export and open the file.