Thank you for your post.
The earliest version of FileMaker Server that was certified to run on Mac OS 10.5.x was version 8.5. That doesn't mean earlier versions won't run, but it could be a possibility.
When FileMaker Server launches, it looks across the network to see if the License Key is being used elsewhere. If the same license key is found, then the error message "Another instance of FileMaker Server is running". However, since this wasn't happening to you the first time, it may be something left over from the initial launch that won't allow you to launch again. Just to be sure, check to see if you still have FileMaker Server running on another machine. If so, stop that server and you should then be able to run it on the local machine.
Do you have access to another machine with an older OS? Try installing and see if it works. If so, then it could be the incompatibility with OS X 10.5.x.
If you want to run under OS X 10.5.x, then you will need to upgrade to FileMaker Server 8.5 or later (the current version is 10).
I think you need to upgrade to the latest filemaker server. I tried to install filemaker server 7 on my mac osx 10.5 machine. It worked but I could not access the filemaker server admin console. By not having access to the filemaker server admin console, I could not change the server paramaters, ie. number of files that the server can host, number of users allowed to access and back up schedules script. Basically server management was not possible.