What kind of dialog Operating system or filemaker?
Check the operating system level permissions on the file.
Check the security settings in the file to see if modify has been restricted for the layout your trying to edit
I think it sounds like a file permissions or locked file issue. Do you get the same message if you try to add or edit a record? Make sure the file and directory are modifiable, as coherentkris suggested. In the old days, I would ask "the file isn't on a CD, is it?" …but that would be pretty rare now.
If it were a FM security issue, I don't think it would give you the option of entering Layout Mode, so I'm pretty sure it's the former.
Check if the file is locked on the hard drive or media there it is stored.
As this is an old file one thing you could try is holding down the Option/Alt key when opening the file. This will bring up the external login dialog and bypass any auto login. Even if the file was set to automatically log in with the default admin/noPassword account that setting may itself be being bypassed in your newer version of FM. I have sometimes found this issue when opening a demo file posted on this forum, and this technique usually overcomes it.