We hope you are having fun playing with the dataset and preparing your submissions.
We noticed there are a few common mistakes/errors which can happen in your submissions.
Hopefully, this list will help you in avoiding them or capturing them early!
1. Multiple bbox for same
track_id in single frame
2. Multiple object class for same
Well, we know quantum computing is cool, and Schrödinger’s cat is too.
Unfortunately, one object can’t have two bbox. Please make sure that you are submitting one
bbox for each
This is wrong because
track_id can’t have 2 bbox.
This is also wrong because the same
track_id can’t be two different objects.
This is correct.
Although I hope bird is at a distance from airplane in above scenario
3. TypeError: Object of type float32 is not JSON serializable
While registering the object using:
register_object_and_location(class_name, track_id, bbox, confidence, frame_image)
Please make sure that values like
confidence, etc are JSON seralizable.
np.float32 should be converted to
This can be done easily in the evaluation codebase, but we don’t want to interfere with your predictions, and cause any unwanted transformation.
4. bbox values
You need to submit the
bbox during evaluation in [
right] format in
5. Hugh difference in online scores than local?
We now report scores for your submission on two of the training flights i.e.
ac23cb93c5c242d2b1bf0633fae9b1e6 in GitLab issue page. (both are present in
You can compare the scores with your local one generated using
core/metrics/ for any potential score differences. This may help in detecting any interfacing mistake early on. Do let us know in case you continue facing different scores.
6. Tip : You can remove
Birds in your predictions
During the evaluation, we are not interested in alerting on birds – so you can remove birds from the results, to avoid unnecessary false positives.
7. Tip : Provide detections with only one class
Helicopter and Airplane can be renamed to “airborne” as we do not assess the correctness of classification. It may help in increasing FAR.
8. Tip : Dataset too big to play around with?
To experiment with fewer flights/images, you can also download a partial dataset (500G) using the
partial=True flag in the
It includes all the frames with the valid encounters of the planned airborne objects. This can be a good start for your first training.
9. Reminder : Timeouts
The timeout for each flight is 600 seconds. (500ms/frame * 1200 frames/flight)
10. Reminder : Office Hours and Help
We are hosting office hours every alternative week on Discord, please feel free to jump in case you want to discuss the challenge with the organizers and fellow participants.
Meanwhile, you can still post your queries 24x7 on Discord and Discussions Forum, and are highly encouraged to do so!
11. Reminder : Using SiamMOT Baseline?
Please note that identical SiamMOT models (with delta <= 1.5% in EDR or AFDR) will be disqualified from winning the prize. An identical model is a model that uses the exact same code and config file provided with the baseline.
12. Hardware used for evaluations?
p3.2xlarge to run your evaluations i.e. 8 vCPU, 61 GB RAM, V100 GPU.
(please enable GPU by putting
"gpu": true in your