a couple of weeks ago a customer were having an issue with the number of queries made against their api. There were some levels they had to set, and it's now on gold level. This will need to be addressed at some point as they may add more checks, or due to other infrastructure limitations will have to reduce the number of queries so I figure planting the seed now would be helpful.
Basic situation boiled down to too many concurrent logins happening at the same time. If a check failed, it was multiplying those logins by 3x. Additionally seemed all the checks seemed to execute more or less at the same time from different locations...i even tried enabling them at different offsets, but any latency or slowness between their checks or our agents updating forced them into a concurrent situation time and time again, causing the api to throw up and the checks failed.
solution would be to have an offset of x seconds for a particular set of scripts. This offset could be determined by a priority level of sorts, where lower priorities are intelligent to know the higher priority items should run first then them, or some sort of specific time offset, and match kind of like you can do in cron. If I create a cron job I can set the minute to */5, which will set the process to run every 5nth minute...if that could be a possibility then could certainly help in similar instances.
Time management with automation can be a pain though, so some thought is needed.