There is no such thing as a "new record mode".
Can you explain what you are trying to accomplish by detecting such? There are a number of ways you might accomplish that result.
One option is to set up an auto-entered serial number field and specify the "on commit" option instead of the default "on creation" option. Then, if this field is empty and you are in browse mode, this is a new record that has not yet been committed.
But it's very easy to commit a record before all data has been entered. It may work more smoothly to set up a special layout of global fields for entering data for a new record. Clicking a button on such a layout 'saves' the data by creating a new record and using set field steps to move the data from the global fields into the corresponding fields in the new record.