Don't make the field accessible in Browse mode on the layout where your limited access users work with this data.
Use a different layout where access is permitted in Browse mode for admin level users. This layout access is something that you can control both through User Interface design and in Manage | Security. Use Manage | Security to make sure that limited access users can't get to a layout where the field can be edited. Use Scripts and layout setup options to keep the from accidentally getting to a layout where they'll see "access denied" or get an error message.
I came up against this issue and can direct "who lands where" right off the bat by using a script that launches automatically upon opening the database controls. It controls which layout the user sees first. Only myself and the owner are directed to the "ADMIN" layout where all fields are accessible.
If [Get (AccountName) = "username1"]
Go to Layout ["layout A"]
Show/Hide Toolbars [Lock; Show]
Else If [Get (AccountName) = "username2"]
Go to Layout ["layout C"]
Show/Hide Toolbars [Lock; Hide ]
Else If [Get (AccountName) = "username3"]
Go to Layout ["layout D"]
Show/Hide Toolbars [Lock; Show ]
and so on...