Service Announcement: Delays in GPU Node Provisioning

Commonsense Persona-Grounded Dialogue Chall-459c12
#1

Update: Evaluators are back online - Service Announcement: Delays in GPU Node Provisioning - #10 by mohanty

Dear Participants,

We are currently experiencing some technical difficulties with the provisioning of GPU nodes through our cloud provider. This issue has led to an unexpected increase in the queue of submissions waiting for evaluation, causing delays in evaluation times.

We understand the impact this may have on your participation in the challenge, and we are actively working with our cloud provider to resolve this issue as quickly as possible. Our team is fully committed to minimizing the inconvenience and restoring normal service levels.

To help us manage the situation effectively, we kindly request that you avoid submitting multiple instances of the same submission for evaluation. This will assist in reducing the queue size and allow us to process submissions more efficiently.

We truly appreciate your patience and understanding during this time. Please rest assured that we are doing everything in our power to resolve the issue and improve the situation.

Should you have any concerns or require further assistance, do not hesitate to contact our support team.

Thank you for your cooperation and understanding.

Best,
Mohanty

Seems like EVERYONE's submission is getting 'FAILED'
#2

Hi, thank you for working so hard on this issue! Is there a possibility of delay in deadline due to this?

#3

Can we try new submissions now?

#4

Disappointing and Arrogant service.

#5

hi @mohanty
The submission page isn’t reflecting new submissions for Task 1. Could you inform when we can anticipate these submissions to be processed and evaluated?

#6

We are still investigating the issues with the cluster, and we will post an update as soon as it is resolved.
In the meantime we appreciate your patience.

We hope to have this resolved by the end of the day today.

1 Like
#7

If the issue remains unresolved by the end of the day, should we anticipate an extension for the deadline? @mohanty

#8

Yes, we will be open to considering a deadline extension if the issue stays unresolved by the end of the day.

#9

Hi, @mohanty
Thank you for your great work and announcement of this isssue.
Do we have to wait until you notify us of which we can submit again?

#10

Dear all,

We appreciate your patience and understanding.

The automated evaluators are back online now.
We will continue monitoring the setup for any further instabilities, but you should be able to make submissions again now.

Regarding the previous submissions, we only requeued the submissions that we had manually stopped. We did not requeue all the submissions as many of the submissions were duplicates, and it would only lead to unnecessarily long queues.

We expect a significant number of submissions over the next couple of days, so we have added dedicated resources for this challenge,
but at the same time we hope you are judicious when making submissions, and avoid making duplicate submissions.

Best of luck,
Mohanty

#11

Dear @mohanty,
Hi! Will there be an extension to the deadline to make up for the lost time? Thank you in advance!

#12

@monumentalcloud : Yes.