The remote evaluator reports the following gRPC error (example from Pokémon Red):
StatusCode.UNKNOWN
Exception calling application: [Errno 2] No such file or directory:
'/app/orak-2025-starter-kit/evaluation_utils/mcp_game_servers/pokemon_red/game/processed_map'
This causes the game server to terminate during environment initialization (load_obs ), and all games are marked as Failed with score 0.
Observations
The error occurs even when the agent code itself is unchanged or minimal.
The processed_map directory does not exist in the repository under:
Creating the directory manually (with a placeholder file) avoids the immediate crash, but it appears the starter kit currently assumes this directory exists in remote evaluation containers.
Environment
Starter kit: latest version from GitLab (as of today)
Execution mode: Mode: Remote
OS: Windows (local), remote evaluation on AIcrowd infrastructure
Python: 3.11 (via uv)
I hope this report is helpful. Please let me know if you need any additional logs or details.
files changed, i also has errors with grpcio, protobuf, numpy, burnycs2 compatibility
since i use conda environment it gives additional headache how to align my environment with constant updates on official server files,
for example server functionality was updated right before NewYear, and it wasn’t announced, so people didnt knew that Metacommands functionality from Research Paper was added, and struggled with old code.
I am currently developing an agent for the Orak Pokémon challenge. I noticed that the remote evaluation environment consistently throws a FileNotFoundError: Errno 2 because the directory evaluation_utils/mcp_game_servers/pokemon_red/game/processed_map/ is empty or missing its required 224 .py files.
To ensure my agent can access the necessary navigation logic and coordinates, I generated these map artifacts locally and performed a git push -f to include them in my GitLab repository.
After this force-push, the map files are now present in the repo, but I’ve encountered two critical issues when running python run.py locally:
Unauthorized Error: The server returns {"detail":"Unauthorized"} even with a freshly generated AICROWD_API_KEY.
Argument Mismatch: My run.py does not recognize the --new-session argument, making it impossible to reset the local session cache manually.
Could you please assist with:
Confirming if the processed_map files should be part of the official starter-kit or if they should be uploaded manually by participants.
Troubleshooting why the repository update might have triggered a persistent authentication failure for local run.py execution.
I think something wrong in the recent submission server.
“Confirming if the processed_map files should be part of the official starter-kit or if they should be uploaded manually by participants.”
→ I don’t think so, it worked automatically in the previous version.
Thanks for bringing this to our notice. One of the recent commits to pokered ended up breaking the maps for pokemon red. It is fixed now. Please let us know if you still face any issues.
