Do you have the Delineate check box selected in Layout Setup...?
I did, I turned it off and I am still getting the same behaviour.
I successfully ran the following test using FileMaker 13 in Windows 7. See if you can spot where it differs from what you did:
Enter layout mode
Double click the Body part label to open part set up.
Select the Use Alternate Row State option
Click Ok to close the dialog
With the body part label still selected, go to the appearance tab of the inspector and select "Primary" from the 2nd drop down at the top of this tab. Select a fill color for what you want for the primary row. Then Select "Alternate" from the drop down and select a contrasting fill color for the alternate row.
Save changes, return to browse mode and view your layout with table view specified.
This produced a table view with the rows alternating in color as I specified as shown in the attached screen shot.
For alternate row coloring to work in Table view, follow philmodjunk's steps above, but you must also change the fill of all your fields to none so that they are transparent. This fixes alternate row coloring in Table view but not active row coloring.
Active row coloring in Table view requires a workaround. Create an OnRecordLoad script trigger to set a global field with your unique ID field. Then set conditional formatting for all your fields to change the fill color when the ID field equals the global field.