Function calling arguments in local_evaluation.py is mismatching with dummy_model's interface

Meta Comprehensive RAG Benchmark: KDD Cup 2
#1

in generate_predictions(),

for query_dict, query_web_search_results in tqdm(zip(qa, web_results), total=len(qa), desc="Generating Predictions"):
    prediction = participant_model.generate_answer(
        query, query_web_search_results
    )

calling dummy_model by (query, query_web_search_results) arguments, and query_web_search_results is a dict type element.

but in models/dummy_model.py:

def generate_answer(self, query: str, search_results: List[str]) -> str:

search_results is a list of str

#2

@vk11 : Thanks for pointing this out. This has been fixed now.

#3

Follow up with an issue when running local_evaluation.py:
when running for line in tqdm(bz2_file, desc="Generating Predictions"):
It throws error OSError: Invalid data stream

#4

@evelynintech : Can you please ensure that you are using git lfs pull to pull the example dataset files via git lfs ?
You can also verify the contents of the example data files in the example_data folder to ensure that they are locally available.

1 Like