Yes, the security is only for the display of data, not the calculation of it, as you have found.
FM made this handy chart:
Notice that all of those security/access actions happen on top of the data layer itself. So things like calculations and summaries that happen at the data layer are not affected by user permissions.
Rather than conditional formatting to hide stuff, why don't you just use FM13's "hide object when" calculation. You could easily hide objects so they are not on the layout at all if the permissions aren't correct.