Thank you for the update @dipam, the new metrics seem more reasonable indeed. I have a few questions regarding the human evaluation though.
Me and my teammate are creating a new persona based chat dataset that is more diverse in terms of conversation settings and the number of persona facts, and will likely result in diverse and factually sensible responses that are a bit further away from the ground truth. Our questions are:
- Is there a ranking cutoff for submissions selected for human evaluation?
- What does the human evaluation metrics / rubric consists of?
- Do we need to include the dataset during gitlab submission or will we be contacted for the dataset, documentation, etc. if our submission is selected for human evaluation?