Thank you for your post.
I apologize for the late reply.
As to the specifics of the issue, I am going to rule out the browser as the cause if you have used that many different ones for now, but it never hurts to clear your browser cache and temp files as sometimes this can cause misbehavior across multiple browsers. I would start by looking at this as either machine or file specific. You mentioned you upgraded to version 11 from 9, and from the wording I am understanding this as meaning you are using FileMaker Server Advanced 11 at this point. Going with that assumption I have some questions about the machine and some steps to try to get to the bottom of where the trouble is occurring.
Starting at the basics, you mentioned you were using an XP 64 system. Is this the machine you are accessing from or is this the host machine? If it is the machine you are accessing from what is the operating system of the host machine running Server Advanced? If this is the host, be advised this is not a tested and certified operating system, which could be a cause. We have seen unexpected behavior running Server or Server Advanced 11 on that operating system. For 64 bit operating systems on Windows we are only tested and certified to run on Windows Server 2008 Standard, so upgrading to that OS might resolve the behavior. For 32-bit operating systems this expands to also include Windows Server 2003 Standard.
Failing this, it could also be caused by file corruption. The easiest ways to determine this are as follows: try closing the file in Server Advanced and move it to a machine running FileMaker Pro or Pro Advanced. Test sharing on this machine via the Instant Web Publishing built into the Pro versions. For integrity of this test, make sure to use a 32 bit operating system, as none of our Pro versions are yet tested and certified on any 64 bit operating systems. If the behavior replicates under these circumstances there is a strong probability the problem is in the file. You can also try recreating the behavior in a new file to verify this. Make a simple reproduction of that part of your file on a few layouts and enter some sample data. Test the new file on both the Server, and the pro machine as described above.
If you determine through this testing that the file is the culprit, create a compacted and cloned copy of the file to see if it behaves normally. To create a cloned and compacted copy you follow an almost identical process for both. Open the file locally (saved to the hard drive on that machine) on a machine that has FileMaker Pro or Pro Advanced installed. Go to the file menu and select 'Save a copy as...'. This opens a standard saving dialog box with one addition. At the bottom will be a third drop down box labeled 'Save a:' that is not present in other programs. Use this to first save a compacted copy and it will automatically rename the new file. Open this new file copy and use the same process, but this time save it as a clone copy. Again a new file will be created and automatically renamed. This will create 2 new copies of your files. The first will be compacted, which resolves a lot of the structural corruption that can occur in a database and cause this among many other types of misbehavior. The second copy will be a copy of the new compacted file but without any records.
Upload both of these files to your server and test the behavior. If both are working normally at this point simply remove the clone and put it in a safe location in case you ever need a backup of the structure of your files. Continue using the compacted copy and rename it to your original file name. If the clone works but the compacted does not then you will need to take an additional step or two to get your data back into the clone. In this case open your original file, and export the records into an Excel file (which I recommend only because of ease of formatting) and review the exported file. Try to manually correct any strange characters or unneeded spaces as much as possible, and once cleaned import the clean Excel file into your cloned database. In theory this should give you the cleanest, most repaired file you can get.
Please keep me updated with any progress, or if you need clarification for anything I covered.
I am accessing from my XP 64 system. I have also used other 32 bit machines in the office and all have the same problem. Please also remember that some list layouts still work.
The server is a Windows Server 2003 SP2 box. I don't really have any say over that machine or updates and things as it also serves a couple of other things.
I did just try to serve the database from my XP 64 machine with the same results. As soon as someone else with FileMaker gets into the office I'll try to serve it there and I'll post if I have any positive results from that.
I've tried to recreate the layout already using the working list layout I have as a template with no success. I've also remove individual graphic pieces from the layouts to see if those were causing the problem. I've navigated to the layout without using any scripts to get there as well but still no change.
Tried the compacting suggestion... no go. Compacted clone... no go.
Okay I finally figured it out. I slapped myself of course. I completely rebuilt the layouts from scratch using the same buttons, fields, graphics, etc. The dang thing worked... So I started comparing the two layouts... On the malfunctioning layouts I had a very large (and completely unnecessary) footer. Once I shortened the footer everything went back to normal. I've actually removed them all together now.
Sometimes when you are making these things you do stuff and forget to fix it. That was the case here. I can't remember why/how the footer got there but there it was. It's clear to me though now that FileMaker 9 and FileMaker 11 handle footers differently. If anything, my stupidity has lead to pointing that out. The default footer size (about the height of a field) seems to work fine. I haven't tested the threshold of that though.
Thanks for your help.