As context can you say whether your adding new content to containers is running client side or server side?
And it's not coinciding with any server backing up or other scheduled operation?
All client side.
When pushing images as quickly as possible with 4 users the country load never goes over 40%.
The only thing the server is need to do is process the data into the database file. No manipulation of the data at all.
Maybe I worry too much. Users are instructed to inform me if any task takes more than 10 seconds. We process tons of photos everyday and this happens rarely enough.
If you use the sort of events performance capture we are working on - and set to collect data only if period > 10 secs - you would soon have enough data to try to establish the pattern?
The remote calls/s and calls in progress spike up as well as the time/call and I think that is leading me to think the issue is with the server processor. But the CPU Load in the activity monitor on the server never seems to climb over 50% even when I try to push it hard with 4 users repeatedly dumping images into the system or running intensive scripts.
What do the "wait time/call" and "elapsed time/call" do when this happens, and the I/O per call.
If elapsed and wait time spike significantly then you have a processor backlog. The CPU itself may not report high numbers in activity monitor but can't take on more work because it is working a thread. Other threads are backed up because of it.
Let us know what you see in that area.