Question: 1. Can Filemaker 11 Server Advanced handle more than 1000 users, any reasonable limits?
This really depends on the database being served and how it's designed. For a moderate database, 250 simultaneous users is reasonable. For 1000 or more users, it's possible, but the solution would have to be well designed.
2. Any comments having a Filemaker Server in a VM?
Make sure the VM is set for only a single vCPU. Even though Filemaker Server 11 (and 12) are multithreaded, the script engine is NOT. I've seen server scripts in this environment bring a server to its knees. Even without server scripts, the overhead of more than one vCPU is more than the benefit of threading processes to another vCPU due to syncing issues. If Hyperthreading is used, a single vCPU will nearly emulate and dual CPU and will perform better, but it's not necessary. Generally when I have any scripts that are intensive, I will put them on a virtual robot machine and have the script run there that will be independent of the vCPU of the FileMaker server. Also raw CPU speed is important in a VM environment. Make sure the hardware the VM server is running on is at least RAID10 or SSD and server class hardware.
3. Changing the application from JDBC ?
Obviously moving off JDBC to all FIleMaker will speed things up.
4. Any concurrent applications?
In a VM environment with only one vCPU only Filemaker server should be running on the VM. If you need another app running, use it on another VM. You will get much better performance this way.
5. Filemaker 5.5 had RAIC system, is it now integrated to Servers or any other way have load balancing.
In a VMWare vHost environment, depending on the version, the load balancing happens on a virtual machine level. This is more for fault tolerance than anything else, but is useful for load balancing the VM between different hardware. To FileMaker nothing is changing when the VM is switching from one node (hardware CPU) to another. Basically it's load balancing VM not the Filemaker processes themselves. Filemaker is never going to get more performance than one core (or nearly 2 for hyperthreading) of a hardware CPU. For other VM servers like Hyper-V or Parallels, they have some of these features, but I'm have had any experience with them in load balancing.