We are happy to announce the launch of the final round of the competition, which involves participants submitting their trained models for all the three tasks.
You can submit your trained models for evaluation by following the instructions in the task specific starter kits below :
For all the three tasks, we will evaluate your submissions on the Public Test Set (v0.3) which has already been released, and a Private Test Set which has not been publicly released yet. During this round, the leaderboards will be generated using the scores from the Public Test Set, and at the end of this round, the final leaderboards (which will be used to determine the winners) will be released which will be prepared using the performance of all the submissions on the Private Test Set for each of the tasks.
PLEASE NOTE that with the launch of the code based submission round, we will stop accepting the CSV submissions, however all your submissions to this round will indeed be evaluated on the Public Test set, and contribute to the public leaderboard of the competition.
For all the tasks, each of the submissions will have access to the following compute resources :
8 CPU
32 GB RAM
1 x NVIDIA V100 GPU
with a total timeout of 30 minutes90 minutes120 minutes for each of the Test Sets.
For reference, the Test Set sizes for all the three tasks are :
If you believe these resource and time constraints will affect your model, please send an email to : mohanty@aicrowd.com, and we will review each of the requests on a case by case basis to assess if we need to revise these resource allocations. In case, the resource allocations are revised, the updated resource allocations will apply to all the submissions.
When will the final time constraints will be fixed? The time constraints will lead to much extra work for many teams. Also, it might be better that code submission can provide how long the code has run. It might be convenient for us that the valid code submission timeout is different from the actual timeout of the instance. For example, we could have a one-hour instance timeout for debugging, but only the output produced in 30 minutes are valid submission.
Could we know what basic docker images are used for code submission, the same from aicrowd-repo2docker 0.11.2?
The public test set phase is available for debugging purposes.
You can run your code end to end on the public split and the logs for this phase will be available, etc. The time taken in image building and public split stages, etc would not be counted towards your 30 minutes timeout (and have their own timeouts to avoid misuse).
The private split run however have 30 minutes purely reserved for your end to end run on the private dataset.
Yes, the default docker image is nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04, however you are not restricted to use that and can choose to use requirements.txt, environment.yml to even custom images & Dockerfile.
Please note, in case you are using your custom image, you need to make sure that they have relevant Nvidia drivers installed to make use of the GPU.
A full detail about the runtime is available in starter kit’s description as well as on Discourse here.
Please check more details in the starter kit’s README and let us know if you still have any follow up questions.
Thanks for your quick reply. I also want to know what is the directory structure in the data folder when evaluating. Do we need to redownload product_catalogue?
The absolute file paths are available as parameters of predict function. (to get rid of any confusion participants may face with directory structures)
We have data, models, etc folders available as examples for the getting started section and a better local development experience. You are free to upload your models and relevant files in folder/structure of your choice. The only constraint is that the entrypoint to your code evaluations will be run.py.
The idea to provide product catalogue by default sounds good.
Please let us check internally and we will update on the request soon.
Hi, is there will be a new product_catalogue or new product_id that we haven’t seen before in private testset?
Since We have done some data augment works based on the product_catalogue;
We are soon releasing some changes to the starter kit, which will allow you to access the product catalogue for each of the tasks. Note that, the product catalogue is the same that is already publicly released, and the private test set does not contain any products which are not already included in the product catalogue.
Hi @mohanty , using git push origin master will not create a code submission? And only use git push origin <tagname> will create a code submission? Is that right?
hi ，I get an error when I make the second submission: Describe the bug
Submission failed : The participant has no submission slots remaining for today. Please wait until 2022-07-03 08:51:17 UTC to make your next submission.
Expected behavior
A Team may make only five submission per task per 24-hour period. Challenge Rule URL.
1.Limited by the current number of commits, I may have only a few opportunities to test my prediction code.
2.I’m not familiar with repo2docker (especially the environment configuration)，it makes me more worried about whether I can finish the competition before July 15th.
Is it possible to increase the number of submissions？
@mohanty@shivam Is the 30min time constraints means that we have 30min to run our prediction code? My submission on task2 always fail without any error message. I think this may cause by timeout, but the time between the failure and the log aicrowd_evaluations.evaluator.client:register:168 - connected to evaluation server is always around 27min, which is less than 30 min.
Hi @mohanty@shivam ，I was trying to make a submission using my own dockerfile. I built the image and ran my test code successfully on my local computer to make sure that my dockerfile has no problem, but after submitted (with debug mode on), it failed. Here’s the full log:
=================
{
“git_uri”: “git@gitlab.aicrowd.com:TransiEnt/task_1_query-product_ranking_code_starter_kit.git”,
“git_revision”: “submission-v0_11_2_1”,
“dockerfile_path”: null,
“context_path”: null,
“image_tag”: “aicrowd/submission:192122”,
“mem”: “14Gi”,
“cpu”: “3000m”,
“base_image”: null,
“node_selector”: null,
“labels”: “evaluations-api.aicrowd.com/cluster-id: 2; evaluations-api.aicrowd.com/grader-id: 70; evaluations-api.aicrowd.com/dns-name: runtime-setup; evaluations-api.aicrowd.com/expose-logs: true”,
“build_args”: null,
“cluster_id”: 1,
“id”: 3739,
“queued_at”: “2022-07-03T20:55:03.753642”,
“started_at”: null,
“built_at”: null,
“pushed_at”: null,
“cancelled_at”: null,
“failed_at”: null,
“status”: “QUEUED”,
“build_metadata”: null
}
Status: QUEUED
…
Status: BUILDING
…
Status: PUSHED
Build succeeded
Trying to setup AIcrowd runtime.
Traceback (most recent call last):
File “”, line 2, in
File “/tmp/evaluator_libs/shared/run_wrapper.py”, line 8, in
from starter_kit.run import Task1Predictor as Predictor
ModuleNotFoundError: No module named ‘starter_kit’
So the dockerfile didn’t fail, “Build succeeded”, it was one of the post-built procedures that failed. I wonder how to tackle this issue. If this is inevitable when using custom dockerfile in order to use custom base images (instead of using aicrowd-repo2docker, which does not seem to allow for specifying custom base image), then what is the correct way to use custom base image?
