First, use duplicate layout to make copy of the layout. Remove the calculation fields that compute your portal totals. Then test this copy of the layout to see if the sum calculations are the bottleneck like I think they are. (A filter on your portal can also slow things down.)
If removing the calculation fields removes the delay, try using summary fields defined in the portal table that compute the same total and place them on the layout in place of the calculation fields.
I've seen claims that they may update quicker. Try it and see.
Another "fix" that is rather drastic and can be difficult to keep accurate totals, is to use simple number fields for your totals that are updated with scripts, script triggers whenever a value in the portal is changed, added, or deleted.