Could you include a "Enabled" checkbox in Create a new check.
So that you can create a check that will not be automatically started every time.
Promoted to a feature! We will get back to you with a more detailed release plan. For now you can set it to Manual to avoid it starting.