Which version of FileMaker Pro are you using?
There is a known compatibility issue with FileMaker Pro 9.0v2 and earlier which prevents Instant Web Publishing from working on Mac OS X 10.5. For FM Pro 9, the solution is to update to 9.0v3. See http://help.filemaker.com/app/answers/detail/a_id/6735
9.0v3 also has a potential problem, with an easy workaround. See http://help.filemaker.com/app/answers/detail/a_id/6766
If you are running FileMaker Pro 8.5, there is no solution short of upgrading to a later version of FileMaker Pro. 8.5's Instant Web Publishing was never updated to run on Mac OS X 10.5. See http://help.filemaker.com/app/answers/detail/a_id/6703
FileMaker Pro 8 or earlier were never officially supported by FileMaker Inc on Leopard, and I would expect the same problem with Instant Web Publishing on at least some of those versions, but never tried it.
A further possible issue specific to Mac OS X 10.5 Server is that the server edition ran Apache 2 as its web server, whereas the client edition ran Apache 1 by default. FileMaker's Instant Web Publishing uses Apache in some versions of FileMaker Pro, and if your version is tied to Apache 1 that might explain the problem.
I am using Filemaker Pro Version 11.0v3