Put the summary fields on a layout that you only use those "couple of times a year". Keep the summary fields off the layouts that you use on a daily basis.
I'm probably being stupid, but the big problem is that on startup or whenchanging tables, the summary(s) automatically runa and delays usage.
Like unstored calculation fields, summary fields "calculate when needed".
If you avoid putting the summary field on any layouts except those where you have to have them (and set up your system so that you only need those specific layouts on those rare occaisions when you need the summary totals) and make sure that your start up script does not contain any references to these summary fields, you won't see these calculation delays.
You can also, in some cases, reduce the calculation delays by being careful to keep the found set on the layout where you have these summary fields very small as the "default" found set. How you manage that can depend on the design of your database and whether you host it over a network or not.