Thank you for the post.
Does the issue occur with FileMaker Server 12 (Advanced) updated to the latest version?
Backing up a file of any significant size is an intensive memory process and will utilize any and all available resources provided by the CPU and RAM. For a computer program to be suffering from a "memory leak" the RAM is both used during processing and then not returned to the operating system once the processing completes.
Please report back with the tests results after updating to FileMaker Server 12.0v4 (Advanced).
PhilModJunk is correct that best practice is to schedule backups during downtime when nobody is using the database file.
Since you mentioned Symantec Endpoint Protection Manager, please note that best practice is to exclude the hosted databases from scans by third party security or antivirus software as this may lead to file corruption.
"If scanning of the files is necessary, please schedule the antivirus software so that it does NOT scan during the time FileMaker Server is backing up the database files. It is preferable to scan the database files after the files have been backed up completely."