I’ve had estimators fail on the smoke test, due to “TIME_EXHAUSTED”, however, the duration of the smoke test is shown as 30,000ms, or 30 seconds while the time limit given here per mlp for this challenge is 60 seconds. Would appreciate an answer on if this is intended or not, and if not, an increase in the smoke timelimit, because I have several estimators that are currently unable to go through evaluation because of this.
You were right that something was wrong, though it turned out not to be a 30 s smoke limit — there isn’t one. I checked every smoke result we have on record and the configured limit was 60 s on all of them. The failures cluster at 60–65 s, not at 30 s.
What was actually happening: your estimator’s setup() and its first predict() were sharing a single 60 s window. The smoke MLP is always the first request a worker handles, so setup time came straight out of the 60 s you were supposed to have for prediction. Across the whole corpus, first MLPs were timing out at 8.4 times the rate of later ones — your submission was hitting a structural problem, not a marginal one.
Setup now has its own separate budget and the 60 s applies to prediction alone. We’re also publishing setup_wall_s on every per-MLP record so you can see exactly where the time went, which was previously impossible — the number was measured internally and then thrown away.
There is a separate 30 s timeout in whest run --runner subprocess that can kill a legal 30–60 s predict locally. It has never affected a graded result, but it can make a local run disagree with a graded one, and we’re fixing it.