Go to Manage Database, Security, Privilege sets, and you will see that you can specify access for each Privilege Set by layout and record. Go to the Custom... on the Records level and you will be able to set field-level access.
Thanks. I knew it was in there somewhere. My eyeballs were just failing me.
The security is set at the table level, not the field level so using Manage | Security will not work unless you store your SSN's in a separate related table so that you can use Manage | Security to control access to the table in which it is defined.
This would be the most secure way to control access to your field so give it full consideration even though this is a fairly significant change to your database structure.
You can use visibility tricks to hide a field, but since this is done at the layout level, it is much less secure than controlling access in Manage | Security. I'd probably use both methods: Manage | Security to be absolutely sure that no one has unauthorized access and then use visibility tricks to hide the field so that users do not see a grey screen hiding the field with "no access" printed across it.
Here's a demo file that demonstrates different ways to hide a field or other layout object from view: https://www.dropbox.com/s/8levaz6deiyzjr8/LayoutObjectVisibilityControlDemo.fp7
Good idea. The <No Access> was looking silly, so I used Conditonal Formatting to hide the field when equal to <No Access>.