It may make sense to have FileMaker only load code signed plugin files (dylib or dll).
So check signature each time FM loads a plugin.
Especially FileMaker could check the signature name for each plugin and report if there is a new plugin found or the signature name changed from old to new plugin.
This way you would make sure nobody just drops a bad plugin into the extension folder or modifies an existing one.
For updates it would not show up as the signature name stays same and signature is valid.