Is it allowed to have some modifications in local_evaluation.py?

Meta Comprehensive RAG Benchmark: KDD Cup 2
#1

Hello,

Are we allowed to make minor modifications to local_evaluation.py say, for example, add one more parameter to some defined function? Please let know.

Thanks!

#2

@soumidas : Yes you can make any changes to the local evaluation script as long as no changes to the current interface of the UserModel class are made.

Please note that on the evaluation servers we do not use the local evaluation script to evaluate the results, and it is provided only for your convenience.

#3

So in the new interface of UserModel, we can access any attribute in batch? In the local_evaluation.py, only return {“interaction_id”: [], “query”: [], “search_results”: [], “query_time”: [], “answer”: []} can be passed to the UserModel. And in the last version of local_evaluation, only query and query_time and search_results can be passed. The evaluation servers will feed with the whole Batch attributes?

#4

@jjplane : Yes, confirming that these 4 items, as mentioned here will be available as keys in the batch passed to the batch_generate_answer function call.