The moment you have all been waiting for is soon arriving! We have prepared round 2, which will soon be starting, we are planning with next week for now. As you are aware, after the excellent results seen in round 1, we have a new and harder challenge this time with a large majority of non-GPS-synchronized sensors thrown into the mix:
In this round, only some receivers will provide GPS-synchronized timestamps while others do not offer such tight time synchronization and thus experience clock drifts or possibly fully broken timestamps. The competitors will have to put efforts into the synchronization of the timestamps of all unsynchronized sensors and the filtering of broken timestamps to get decent results. However, they can improve their localization algorithms by incorporating the barometric altitude information.
We have also listened to your feedback and our own experiences, so we made a few tweaks, where possible. Let me summarize them:
-
The coverage requirement of 50% was unnecessarily low in round 1, contestants still had very similar results with 80+% of the data meaning this led to optimizations that were not desirable. However, correctly picking and discarding outliers is still a potentially important task, so we set it at 90% this time. We toyed with the ideas of penalties for non-predicted records but this is very hard to calibrate in an unknown setting like ours.
-
We cannot prevent the use of future data in this competition setup (and for some of our applications we do not need/want to), so we have no rule against it in this round. We are thinking about coming up with a new approach in this direction for the future, ideas and input welcome!
However, we did implement a hidden data evaluation approach to mitigate overfitting on the test dataset. The score on the scoreboard will now be calculated on a fixed, arbitrarily chosen, 30% of the 300 aircraft tracks that need to be predicted. The scoring on the full dataset will then be revealed after the end of the round. -
We have limited daily submissions to 5 (excluding failed submissions), this should be more than enough. In particular considering the (necessarily) increased size of this now complete and authentic OpenSky dataset.
If I have missed anything super important, let me know! I am looking forward to round 2 with new and old contestants. We should publish the datasets very soon.
Edit: Training and competition datasets for round 2 can be downloaded now! Happy localizing!
Best,
Martin & the rest of the organizers