Looks like when we reset DerkEnv we generate a new item set everytime. Which is good.
But I’ve found that my bots sometimes didn’t get all 3 items.
Is it possible during evaluation? If not I believe it should be fixed. And I think it should be configurable in the DerkEnv.reset method. So every reset I can choose what items to use.
1 Like
I don’t know about not receiving all 3 items, but I have noticed that the random generation will sometimes result in a really unfair game.
I asked on discord if the evaluation score was averaged over a large number arenas, which might mitigate the issue, but never received a response.
I agree that the competition would be better and the problem much more interesting if you could choose the items (ideally learning to choose the items). The environment supports that (not on reset though).
Hey, we will discuss about allowing the participants to choose items with the team and get back to you. With regards to the evaluation score yes it is an weighted average score. Your bot will play against each of the 3 evaluation bots 128 times each. The score against that particular bot will be the average of the 128 runs.
Possible solution is to remove very weak combinations.
Left only 10-20 possible combinations or something like this.
That’s a good approach too and could be really good if the combinations where designed well.
Does the current randomization allow duplicate items?
If the items are chosen without replacement, currently there are 15 choose 3 = 455 combinations per agent.
But then the problem depends on what is given to your whole team and then what is given to the opposing team…
If freely choosing items I would hope there would not be some really dominate combination.
Reviewing observations after reset I saw situation where one of my units had only one item.
Also I saw they had same items at least on the tail.
I am not sure if you can determine the items given visually.
According to the documentation there are 7 items for the arms slot, 5 items for the misc slot, and 3 items for the tail and some of these can be empty (not sure if the random selection does or not), but definitely shouldn’t have more then one item per slot.
What I am concerned about is the training time and network capacity required to learn all these capabilities. That is going to put this competition out of reach for the average person without a big GPU and/or computing time, which I really think is not in the spirit of this competition.
I think empty slots are bad idea.
Removing empty slots will solve half of the issues.
The most recent version of gym-derk was updated on 1/21, I assume, in response to this issue. It tweaked how the items are assigned randomly. But the details are now in the documentation.
Why wasn’t there any communication with us? This appears to be a pattern.
Hey @bwitherspoon,
Indeed, the gym-derk was updated to respond to this issue. We do hope we keep improving as we keep getting more feedback 
This update to gym-derk in our evaluator was not made reflectively. We were working on a couple more improvements over the past few days (GPU-enabled submissions, access to build and run logs, etc) and are releasing all the changes together 
(You can check out the updates here [Challenge announcement | GPU submissions, build & run logs, and more])
Cheers,
Yoogottam
2 Likes