Hello admins, can you tell us if anything about private data has changed since day 1 of phase 2? This is important to know for reproducibility and experiment evaluation.
- On day 1, were private nonegocentric images being resized incorrectly and then later fixed?
- On day 1, were private web search indexes corrupted and then later fixed?
- Since day 1, has 10-20% more chunks been added to private web search corpus?
The above 3 occurred on public data, so I’m wondering if they also affected private data. Additionally were there any other changes to private data and/or submission process that has changed since day 1 of phase 2?
Based on comments by @yilun_jin8 here, here, here (and displayed in quote below). It seems that the private LB test data has been updated. Does this mean that everyone needs to resubmit all their submissions again to see the improvement of the new v0.6 private web search databases?
Can admins please answer the 3 bullet points in my previous post? And in the future, can admins please inform all partcipants when changes are made to private data and/or submission processes? Thank you!
Because the size of the search indices expanded dramatically after the latest update from Meta, there were lower than expected storage space on the evaluation nodes for participants to store their models. Therefore, you may see that your submission failed due to limited storage.
We have addressed the problem by increasing the disk space on the node. There will be at least 250GB for participants’ code and models.
We have also re-queued all recent submissions to account for this, and also for the latest updates in the search index.
Hi @Chris_Deotte ,
You don’t need to re-submit by yourself. We have queued all previous submissions for re-evaluation, and the re-evals will happen automatically.
Members from the meta organizing team will answer your questions about the data and indices.
@yilun_jin8 why the submission in Jun 2 hadn’t re-submit ? I also met disk space problem. Are they in the waiting queue?
@beam_search We temporarily suspended the re-evaluations until the fix for search index is addressed. We are testing the new index that meta has shared with us. We will continue to the re-evaluations once we are sure that everything works as expected.