Deadline Extended to Feb 8 + Instructions for Final Submission Package

Orak Game Agent Challenge 2025-06ec83
#1

Hello everyone,

The challenge deadline has been extended to February 8 (to accommodate recent server downtime). Please make your final submissions by then, and start preparing your final evaluation package now. Each team may submit one final submission per track, and the final agent must be runnable with the Orak framework and compatible with 4 games.

Please ensure all eligible entries are submitted with a complete final submission package. All previously shared requirements still apply, including:

  • Model artifacts and documentation
  • Runnable agent code
  • A 2-page design and training PDF
  • Reproducibility artifacts
  • Evaluation summaries plus required metadata

Please refer to the full list over here.

How to share your artifacts

Models:
Models must be shared as gated Hugging Face repositories and you must grant access to:

Gated model setup instructions:

Code:
The remaining code should be shared via a private repository on https://gitlab.aicrowd.com, with appropriate access provided for the organisers. Please grant GitLab access to: @aicrowd (AIcrowd · GitLab) & @orak-krafton-eval (orak-krafton-eval · GitLab)

Edit: If you are unable to locate @aicrowd please grant access to @aicrowd_team and @aicrowd-evaluator.

Track 1 model size clarification

As clarified before, 8B models are allowed even if the actual parameter count slightly exceeds 8B (for example, Llama-3.1-8B). If you use multiple models that share identical base weights, the shared base weights should be counted once, and any adapter or LoRA parameters should be counted separately.

All the best,
Team AIcrowd

#2

by “Model artifacts” you mean that someone had fine-tuned 8B model? which is very expensive.

#3

Quick question — for agents using an open‑source 8B model via API (no weights hosted), what do we need to submit?
Is a gated HF repo still required, or just docs + reproducibility details?

#4

For Track 2 (who might use commercialized APIs without fine-tuned),

can we skip to submit Model artifacts and documentation, Reproducibility artifacts?

calling OpenAI API is all

#5
#6

Is OpenAI gpt 40 mini model allowed for Track 1?

#7

@aicrowd_team I cannot find the @aicrowd name in GitLab’s “Invite a group” window.

#8

Hi @ChoiSoojin We have been added to the GitLab and received the notification for the same.

#9

we using this model,

Qwen3VL-8B-Instruct-Q4_K_M.gguf
still dont understandable what need to be done,
this model is not fine-tuned and we only use it for local inference for prompt.
How you expect it be stored on gated Hugging Face repositories?
i think you doesn’t want us copy to huggingface,
because this model is already open and everybody can download it and deploy to endpoint.

#10

cannot find aicrowd only aicrowd_team exists
image

#11

Clarification on final evaluation / “model artifacts”

For the final evaluation, KRAFTON will run your submission on our system (including hidden tests). To be eligible for final rankings, your submission must be fully reproducible end-to-end.

  • If you fine-tuned or modified weights (LoRA/adapters/merged checkpoints/custom weights), you must provide those artifacts.
  • If you did not fine-tune, you don’t need to upload base model weights—but you must provide exact model ID + revision, exact format/quantization, and all inference/runtime details (prompts/templates, decoding params, configs, scripts, any extra files/data) needed to reproduce the result.

If we can’t reproduce your results due to missing details/artifacts, the submission won’t count toward final rankings.

1 Like
#12

our project is here. if you can’t see ours due to the issue. let me know.

#13

there is no @aicrowd in the gitlab.

image
image508×604 49.2 KB

#14

Hello all,
If you are unable to locate @aicrowd please grant access to @aicrowd_team and @aicrowd-evaluator.
Thank you