Dear all,
Round 1 has been completed.
To compete in Round 2, the requirement was to be in the Top 20 with a valid submission.
Given that we have less than 20 teams on the leaderboard, I am glad to announce that all people who have submitted can now partecipate in Round 2!
Final leaderboard for Round 1:
1) CIIRC-Incognite
2) mrrobot
3) robot-ea
On the other hand, we expected a wider partecipation (given that 158 people subscribed to the competition) so we will probably send a survey to all subscribers to understand why only a few subscriptions turned into submissions.
One of the main reason is probably that this challenge is very hard and complex to tackle!
There are many problems to face at once, such as:
Learning to properly abstract the environment
i.e. recognizing objects in some way
Building models of the world
i.e. learning how the state of the environment evolves by acting on it
Learning reliable skills
i.e. learning appropriate actions that consistently achieve certain states
Planning
i.e. how to reach extrinsic goals by applying the learned skills
… and all these pieces of knowledge interact with each other and must be learned together.
Indeed so far the highest score (0.235) has been achieved by controllers which just stand still, since in many of the goals that we test it is advantageous to just leave objects as they are rather than pushing them around randomly.
An agent doing random movements scores about 0.100 instead.
I would like to make a mention here for teams AutoLearingMPI and isi who got their scores without resorting to these two baseline controllers.
isi team also got up to 0.134 in the latest submission (better than random), altought it was after the deadline had expired.
Moving on to Round 2!
So we have one month left to improve our controllers and beat both the random and the static controllers!
What are your expectation?
Will you beat the random and static controller?
What have been the main challenges for your team so far?
Round 2 will soon open.
Good luck and good to everyone for this last month of challenge!
For our team, the task was challenging even with GT masks, and the best learned masks that we’ve got were obtained with pre-trained model (using ImageNet).
So given that in Round 2 both of this options are not available, and in addition, we would need to train on different hardware, we are not sure that it would be possible to fix all the problems in 2 weeks.
Maybe it would be reasonable to make Round 1 rules the final rules as the problem was already challenging enough (e.g. no one gets better than “Doing nothing” policy).
Also, it would be beneficial to know the prizes for winning places.
Maybe it would be reasonable to make Round 1 rules the final rules as the problem was already challenging enough (e.g. no one gets better than “Doing nothing” policy).
Indeed the challenge proved to be too hard.
However, changing now the rules and allow any solution would not be fair to those who have been trying to achieve it following Round 2 spirit.
Instead, we can keep two classifications and allow people to submit both solutions that follow Round 2 rules and those which would be valid only under Round 1 rules.
A single team may have different submissions following either ruleset and will be ranked in both.
Also, it would be beneficial to know the prizes for winning places.
Initially, we had planned to have some travel grants to NeurIPS as prizes but we weren’t able to provide them in the end.
So the main prizes are the recognition and glory
Talking about recognition, we will have a 20-minutes presentation of the competition @ NeurIPS on December 13th.
If nobody gets any significant result (i.e. above the no-action baseline) I will probably focus on the challenges of the competition and how hard it is, mentioning some possible approaches.
However, if we get a result above the baseline, I will devote a part of the presentation to that winning approach above the baseline (with preference to those who followed Round 2 rules).
If the authors of that solution are present at NeurIPS and are willing to explain their solution in person, I think we maye be able to arrange so that they present that part.
We have been told that there will be a volume of Proceedings of Machine Learning Research (http://proceedings.mlr.press/) associated to NeurIPS competitions, so we will recognize notable solutions in there as well when describing the results of our competition.
