Thanks for posting.
FileMaker Server 10 is not certified or tested on Windows 2011 Small Business Server. The currently supported Windows environments for FileMaker Server 10 are Windows Server 2003 Standard SP2, Windows Server 2008 Standard SP1, and Windows Vista Business SP1. Information on FileMaker Server 10 system requirements can be found at http://help.filemaker.com/app/answers/detail/a_id/7152.
Other users have reported that they can use Small Business Server in earlier Windows Server versions after SharePoint is removed. Windows 2011 Small Business Server looks to be based off of Windows Server 2008, which is supported for FileMaker Server 10. The main issue with Small Business versions is they come with services such as SharePoint, Exchange Server, and sometimes SQL Server that can cause conflicts with FileMaker Server. Disabling these extra services may get FileMaker Server to run correctly.