FileMaker released an update of version 12 to achieve compatibility with Mavericks. Not so for version 11.
Running OS X 10.9.1 Mavericks on my MacBook Pro mid-2009 8 Go RAM, i was able to use during days FileMaker Pro Advanced 11.0v4 on a professional environment without any problem.
11.0v4 was a special release for OS X 10.7 Lion. What is your exact release of FileMaker Pro 11 ?
Did you try to uninstall and reinstall it ?
I did it because i installed Mavericks "from scratch" on my MacBook Pro, without any problem too.
Bonne chance, Fred