If you don't already have it, get FileMaker Advanced so you can use its developer tools to work with your database.
One of my files has now reached nearly 4.5GB in size and I have not had any trouble with it. Granted, this is an "archive" file where we park invoice type records no longer needed except for long term summary reports and such (Active data entry takes place in another file), but it does well for what we ask of it.
Often for a file of that size and age, there could be any number of portions of the system that are no longer needed and carefully removing them can produce a smaller and easier to maintain system.
The Database Design Report is one tool you can use to check and specific fields, tables, layouts etc are actually made use of in the current system. It's my understanding that there are also a number of third party developer tools available and you may want to research them to see if they offer any additional assistance in managing this file.
Yea the Database Design Report helped emensly, espeically since I have absolutely no contact with the person that designed the database (no one else knows anything about fmp or the sever). To make matter more complicated, several executable files were developed by other programs and incorporated into the DB.
I suppose it will be a few months before I have everything nailed down.
Thanks for the comment, Phil