Turns out setting Warn & Error thresholds doesn't work on these checks since they are interpreted as Warning & Erroring if they go ABOVE the value you enter in the UI.
But the returned value for this type of check is an amount of days rather than a performance stat so you can't seem to achieve the desired effect.
Example: I want to be alerted as a warned after 21 days and produce an error alert after 14 days. This is not achievable in the current way the Error/Warning thresholds work so only a fatal alert is possible.
This is something that we will take a look at, you should be able to use error and warning thresholds.
Let me get back to you when we have a planned date for this.