Missing processed_map directory causes remote evaluation failure

Orak Game Agent Challenge
#1

Hello Orak 2025 Organizing Team,

I would like to report a blocking issue I encountered when running the latest Orak 2025 starter kit in remote evaluation mode .

Summary

All games fail immediately during remote evaluation due to a missing directory:

/evaluation_utils/mcp_game_servers/pokemon_red/game/processed_map

Error Message

The remote evaluator reports the following gRPC error (example from Pokémon Red):

StatusCode.UNKNOWN
Exception calling application: [Errno 2] No such file or directory:
'/app/orak-2025-starter-kit/evaluation_utils/mcp_game_servers/pokemon_red/game/processed_map'

This causes the game server to terminate during environment initialization (load_obs ), and all games are marked as Failed with score 0.

Observations

  • The error occurs even when the agent code itself is unchanged or minimal.
  • The processed_map directory does not exist in the repository under:
evaluation_utils/mcp_game_servers/pokemon_red/game/
  • Creating the directory manually (with a placeholder file) avoids the immediate crash, but it appears the starter kit currently assumes this directory exists in remote evaluation containers.

Environment

  • Starter kit: latest version from GitLab (as of today)
  • Execution mode: Mode: Remote
  • OS: Windows (local), remote evaluation on AIcrowd infrastructure
  • Python: 3.11 (via uv)

I hope this report is helpful. Please let me know if you need any additional logs or details.

Best regards,
Goldi