💬 Feedback & Suggestions

Orak Game Agent Challenge
#1

We are constantly trying to improve this challenge for you and would appreciate any feedback you might have! :raised_hands:

Please reply to this thread with your suggestions and feedback on making the challenge better for you!

  • What have been your major pain points so far?
  • What would you like to see improved?

All The Best!

#2

Hi snehananavati, my team and I are having some doubts.

  1. Is a team available to join and post a chance to win in both tracks?

  2. May I know the sponsored credits (NVIDIA Brev credits, AWS Bedrock credits, OpenAI API credits) will be distributed to the team? If yes, when will it being distributed? How many credits will be distributed to each team?

  3. Is there any other ways to access the starter kit as we clicked the link and got a 404 error.

We are looking forward and appreciate for your help! Thank you!

1 Like
#3
  1. Have you established a Discord server or any other platforms to facilitate communication among everyone?
  2. How are bonuses distributed? For people from different countries, the distribution seems to be quite cumbersome.
  3. Is participation limited to Koreans, or is it open to everyone?
#4

Hello @cheong_wei_xun
Details on compute credits and how to claim them can be found here: 🪙 How to claim compute credits

#5

Hello @vk_c,

  1. You can use this Discourse forum for all queries and feedback. Additionally, we have a Discord text channel for collaboration: AIcrowd
  2. All the details on compute credits, how they are distributed, and the process to apply for them can be found here: 🪙 How to claim compute credits
  3. The challenge is open to all individuals.

All the best!

#6

Hey
How do we know our submission is evaluated. it is showing “evaluating” for quite some time. when will the leaderboard be updated

#7
  1. Difficult initial setup
    The initial setup is quite difficult for testing.
    Should we simply follow the instructions on the starter-kit page to see the first test?

Steps such as:
(1) Cloning the starter kit
(here, I had to install diambra)
(2) Running diambra run --path.roms executables/streetfighter3/roms and then uv run python run.py --local

However, the folder does not contain any ROM files.

After cloning the starter-kit GitHub repository, Super Mario runs successfully,
but the other games get stuck at “launching.”
For example:
uv run python run.py --local --games pokemon_red

  1. DIAMBRA
    I’m sorry, but I don’t have any prerequisite knowledge about this framework.
    Do I need to understand DIAMBRA in order to participate in this contest?

  2. Sample code
    There is no explanation of how the LLM is connected to the game and where can I see the interaction between LLM and game in the code.
    Providing this would be very helpful for understanding where to begin.
    I’m not sure whether the ORAK benchmark paper includes this information, but at least the starter-kit page does not seem to provide it.

1 Like
#8

a Tutorial video will be very helpful
(1) Starter-kit setup
(2) Simple agent development process for one game
(3) Result submission

#9

Agree, there is should be more detailed tutorial of what we need for games, like ROMs or additonal files. And where we have to put all of these.

#10

Hello all, Thanks for the feedback. We’re reviewing it and exploring how we can better support you with the submission process.

#11

I get this error when trying to run run.py locally

    from agents.openai_starcraft import OpenAIStarCraftAgent
ModuleNotFoundError: No module named 'agents.openai_starcraft'
#12

check this file:
agents/config.py
there is actually no prepared script for sc2, so you can use random one for tests
StarCraftAgent = RandomStarCraftAgent
or create another that is actually your competition goal

1 Like
#13

I’ve been wondering:

For Track (1), with the limitation of LLM size:

  • Full parameter tuning with 8B: ACCEPTABLE (total size remains 8B).
  • LoRA fine-tuning with 8B: NOT ACCEPTABLE (total size may exceed 8B).

is it right?

#14

I have to submit one finalized LLM model to cover five games.
So, independent fine-tuning for each game is impossible.
ex)
(O) LoRA_for_ALL for five games.
(X) LoRA-1 for supermario, LoRA-2 for starcraft

am I right?

#15

Hello, im late participant trying to register , but it seems,https://www.diambra.ai/register is not awailable already, is it correct?

Page Not Found

The page you are looking for doesn’t exist or has been moved

#16

the url is broken.

but, you can access and register.

1 Like
#17

Hi @seokwoo_song Please refer to this post for clarification on model limits: Clarification on Track 1 “8B parameter limit”