This is an issue with server 10 and 11 that crops up occaissionally. FileMaker Inc. is aware of the issue but has had great difficulty in reproducing the issue let alone resolving it.
For Server 10, the only fix I've found is to kick every one off the network, (and you can't use Server Admin for this, you have to go around and make sure everyone is out of the database files), Then restart the server's computer. This is the equivalent of a force quit of your database so FileMaker Server will first run a consistency check on all hosted files before reopening them. Once you've restarted the computer, everything backs up again until the next time the server "hangs" on a scheduled back up.
I did pretty much exactly that. Looks OK now. Running a backup to be sure.