There's documentation that comes with FileMaker server advanced that lists the ports that should be open. I'd review that documentation and confirm that each port specified is open.
It appears that Filemaker Pro and FMS may have not been the culprits, considering the traffic in question is internal. The likelihood could be DNS related, not completely sure because of how enabling one disables another. It seems to be working now, so I'm going to go ahead and consider this resolved for the moment.