Have you checked the event log within the Server operating system and FMS 12 server log for any clues?
What Jeff is saying is where to start: what do the logs have to say.
In addition, how is your server configured:
Is verification turned on for the backup?
The files are large (~20 GB total for all of them),
What is the largest file size?
The server is up to the required specifications in hardware and software.
Please post details, the requirements don't tell us much in relation to what your solution is and what it does. make sure to include the free disk space on all drives on your server and indicate where live files and where the backups go.
Also let us know if you have Progressive Backups turned on.
The server is dedicated to Filemaker (both FileMaker Server 12 and FileMaker Pro 12 are running on it).
Ay, that's not good. I hope you're saying the FMP 12 is just installed but not actually "running" on the server.