FileMaker databases should be hosted by using either FileMaker Pro or FileMaker Server to open the database on the server. Then users with FileMaker installed can use Open Remote to access the database. If you use FileMaker Pro to host the databse, it will support a maximum of 9 other users connecting to the database so "up to 10" puts you right at the limit to how many users you can support before you need to upgrade to FileMaker Server in order to support a larger number of users.
If you elect to use Instant Web Publishing, your users can use web browsers to access the database, but FileMaker pro limits you to 5 such connections at any given time. FileMaker Server Advanced can also be used for Instant web publishing to support access from a larger number of Web clients.
SBS2008 was not listed as a system for FileMaker Server 10 and I forget whether or not it's listed for Server 11. You can reasearch system requirements in the knowledge base (See link at top of this screen) to get more info on this. As I understand it, SharePoint--a part of SBS conflicts with server (at least server 10) and that you end of needing to disable this part of the system to get FileMaker Server to work--though this might only apply to web publishing so please check on this further. WHen I researched this for Server 10, I was able to call FileMaker tech support directly and ask them so you might want to do the same. A server machine with WIndows Server installed, did not have this issue.
There were problems running SBS2008 with FileMaker Server 10. I had to scrap my SBS server install and go with Windows Server 2008. I don't have any experience with 11 but I do think that the problem had to do with the bundled services included in SBS2008 so I would tend to think it's still not a good idea.