You could have a menue-file, a globals file, that checks for the number of clients (get function in a script). If more than 5, give a message and quit FileMaker
AFAIK, there is no native mehod for this. Count of clients is per file, therefore a global- or menu-file that stays open
There is Get (UserCount), which might be useful. I can't think of any other information FileMaker Pro is able to access about the connected users. FileMaker Server is keeping track of concurrent connections including the who/when/where for the current live connections, but we as developers have no access to that information within FileMaker Pro.
If you use Get ( UserCount) and bounce the n+1 user, you'd be somewhat limited (as is FMServer) to saying simply, "The maximum number of clients is already connected to this system please try again later." and booting them off the connection.
I'd probably roll my own licensing/login module. If you already keep a table of user accounts to assist in managing user-specific settings and preferences, you'd be half way to accomplishing a scheme that will be able to inform the n+1 user not only that the connection limit has been exceeded but also show them an 'error' layout that does not allow them access to any other parts of the solution. (And has some helpful text on telling them to log out of their current session, which is holding up others, and maybe even help them know who forgot to log out on which device.)
In other words, let them log in, don't let them do any work, and give helpful guidance on what to do next.
Both a one-record table with at least some non-global fields (and thus live data is accessible to all users logging into a solution) and maybe a home-grown access log table (to register the date/time/ip/device/user name/etc. for each log in) would seem to be useful elements in solving your problem. I've done that for other clients, and is not all that difficult to build.
The trickiest part is when users fail to log out (especially easy on iOS), thus blocking a connection. For that, you build On Timer triggers and auto log out processes throughout the system to force the connections to close, or at least open up an available connection.