Definitely not an al or nothing limitation. You should be able to have portal row button hide on a row by row basis provided:
1. The button is actually in the portal row and not hovering above the portal—check this by (a) making sure it is small enough to entirely fit within the portal row, and (b) giving it a nudge up then down to make sure it centres on other portal objects.
2. The hiding condition is focussed on the individual portal record—this is perhaps more likely where you are going wrong.
It was one of those; the latter I think since I tweaked both! A tip for people; magnify portals when buttons are 'tight-fitting'. I'd thought I'd been careful to fit everything inside, but the centering advice should be paid attention to.