Hello,
When running my estimator locally with whest smoke-test, whest validate and whest run, everything works, but when i try to submit it either via web or CLI, i get this cryptic error message:
participant subprocess crashed: unexpected frame type 'result'; expected 'predict_start'
Locally, i use whestbench 0.12.0rc0, but on the submission page, tab configuration, i can see that the server seems to be using version 0.11.0rc0
Can you advise me on what could be the problem here ?
Thanks
Thanks for reporting this @ashen.l
This issue has now been fixed.
Previously, this could happen when an exception was raised inside the setup() call. The original traceback was being overwritten by a much more cryptic internal error:
participant subprocess crashed: unexpected frame type 'result'; expected 'predict_start'
With the latest fix, the actual underlying error should now be shown instead, making these failures much easier to debug.
All affected submissions have been re-evaluated as well.
Many of the earlier failures we saw were due to flopscope-client not yet supporting some native flopscope operations. We are continuing to monitor submissions closely and will be pushing a few more fixes soon to improve parity between flopscope-client and flopscope.
We are also working on additional tooling that will let you locally use the same interface that the evaluation servers use. We expect to roll this out by the end of this week.