Specify the following formula for the conditional formatting on your field:
Length ( your_Table::your_Field ) > 10
Then click the "More Formatting" button in the Conditional Format dialog box and select the desired font size.
If the field has more than 10 characters, the font will then be reduced to the font size you've set.
You can change the "10" in the formula to whatever number of characters you want.