I think it depends on how the container layout object is formatted.
On the 4 tab of the inspector, you can set a container object to be optimized for Image or for Interactive formats (pdf…)
If you chose the latter, then with some formats such as PDF, you can't replace the file (no access to the native right-click menu because it's overridden by the PDF readers'. You have to remove the file (using the Delete or Suppr key for instance) before you can insert a new one.
Not sure if this is the situation you ran into though.
And OLE is no longer supported in FileMaker. So Insert Object is no longer possible.