Use File Options to set up the file to automatically log the user in with a very limited account such as one with a read-only privilege set that only allows access to your main layout. Set the same file options to specify the main layout as the first layout seen when the file opens. Use File Options to perform a script OnFirstWindowOpen (a script trigger).
Have this script use Re-Login to open the FileMaker password log in dialog. You can set up this script with an If step immediately after the relogin script that uses get ( LastError ) to check to see if the user successfully entered an account name and password or not. You can put this in a loop where successfully entering log in credentials exits the loop and failure pops up a custom dialog telling the user that they entered the wrong password and then the scripted loop repeats to re-open the password dialog.