That button is for borders between repetitions of the same field. It is not for portal row borders.
Assuming that this is FileMaker 13 (or possibly 12),
Enter layout mode and select the portal.
In the Appearance tab of the Inspector, select Portal: Row from the upper drop down instead of portal. Then select the normal border settings.
And you can specify this three times: Once for Primary, alternate and "active".
Thanks a lot for your quick help. I got confused when I saw this button didn't work... Your solution worked like a charm !