Hello admins, can you tell us if anything about private data has changed since day 1 of phase 2? This is important to know for reproducibility and experiment evaluation.
On day 1, were private nonegocentric images being resized incorrectly and then later fixed?
On day 1, were private web search indexes corrupted and then later fixed?
Since day 1, has 10-20% more chunks been added to private web search corpus?
The above 3 occurred on public data, so I’m wondering if they also affected private data. Additionally were there any other changes to private data and/or submission process that has changed since day 1 of phase 2?
Based on comments by @yilun_jin8here, here, here (and displayed in quote below). It seems that the private LB test data has been updated. Does this mean that everyone needs to resubmit all their submissions again to see the improvement of the new v0.6 private web search databases?
Can admins please answer the 3 bullet points in my previous post? And in the future, can admins please inform all partcipants when changes are made to private data and/or submission processes? Thank you!
Because the size of the search indices expanded dramatically after the latest update from Meta, there were lower than expected storage space on the evaluation nodes for participants to store their models. Therefore, you may see that your submission failed due to limited storage.
We have addressed the problem by increasing the disk space on the node. There will be at least 250GB for participants’ code and models.
We have also re-queued all recent submissions to account for this, and also for the latest updates in the search index.
@beam_search We temporarily suspended the re-evaluations until the fix for search index is addressed. We are testing the new index that meta has shared with us. We will continue to the re-evaluations once we are sure that everything works as expected.
@Chris_Deotte private web search index was not corrupted and has been working as expected since day 1. We did add 10-25% more chunks to private web search corpus but that only affected the validation / public-test queries for consistency. The web search recall for private set remains the same.
I am confused. On day 1 of phase 2, wasn’t the private web search using v0.5 and then recently it was changed to private web search v0.6 (with 10-20% more chunks)? Wasn’t that the reason that private web search started returning None and the disk space on server starting running out?
So in other words, didn’t the private web search change during phase 2? And solutions submitted during day 1 with v0.5 will perform differently on leaderboard than the same solutions submitted today with v0.6?
i.e. if nothing changed with private data, why are we re-running all submission to the leaderboard?