💬 Feedback & Suggestions

Brick by Brick
#15

Response from the organisers:

That’s correct—the 20% and 45% values do not refer to the number of “rows” or chunks. Instead, they refer to the proportion of time series allocated to the private, public, and secret sets during dataset preparation, as described here: AIcrowd | Brick by Brick 2024 | Challenges

Specifically, during the preparation stage:

All data from the three buildings are combined into a single dataset and then segmented into distinct sets for training, leaderboard testing, and secret competition testing.

In this step, approximately 20%, 45%, and 35% of the time series are assigned to the private, public, and secret sets, respectively. However, note that the length of each time series can vary. Later, the dataset undergoes further processing which yield the “rows” and “chuncks” available to the participants:

Time Series Chunking: The dataset is further divided into shorter segments or chunks with durations ranging from 2 to 8 weeks.

I hope this clarifies your concern.

#16

Thank you, this makes sense!

#17

Hi,
When will Round 2 leaderboard open? Submissions currently just say submitted but scores aren’t being shown.
Thanks!

1 Like
#18

Update: the issue is now fixed.

1 Like
#19

hi @snehananavati, are we not allowed to form/join teams in round 2? it says team freeze in effect

#20

Yes, the team freeze has been in effect since 10th January. It typically begins three weeks before the challenge deadline to prevent individuals from submitting duplicate entries through multiple teams.

#21

oic, thanks for the clarification!

#22

Could you please clarify the evaluation metrics used for scoring? The competition description mentions “F1 micro”, but the description of it ("This involves computing precision and recall for each label) matches the definition of “F1 macro”.

F1 micro typically means calculating metrics globally by counting the total true positives, false negatives and false positives.

Thank you!

#23

Hi Thomas,
Thank you for pointing that out. You are correct—the metric used is Macro F1.
The typo has now been corrected.

#24

Thanks for clarifying! Please note that the description still contains references to “micro”:

image
image896×362 49.7 KB

Not a big deal, just FYI

#25

Hi, I have a question I’d like to ask: Can the final winning team participate in the workshop remotely?

#27

Hi there,

We would like some clarifications on the labeling structure.

The description says:

Labeling Structure

The labelling adheres to a modified version of Brick schema version 1.2.1, featuring 94-point sub-classes. Each data point is classified with multiple label types:

  • Positive Labels: The true label and its parent classes.
  1. What is that modified version of the Brick Schema?
  2. What are the 94 point subclasses? Or how does that correspond to the 240 classes of the train_y csv? Are classes with a child not counted into these 94 classes (i.e. the 94 classes are lowest level)?
  3. If all the datapoints are subclasses of Point and the parent classes are labeled as True. Why is the Point class in train_y -1 for all samples? Shouldn’t it be 1 because it is the parent class of everything?

Thanks for the clarifications!

2 Likes
#28

Hi @mike_q

Remote presentations will not be facilitated by the conference, so all attendees are expected to present in person. Each participant is responsible for their own conference registration. For those whose papers are accepted – typically among the top submissions – their registration will include the paper publication fee. This process is similar to submitting a paper to a conference and having it published. The competition organisers undertake this work on a completely voluntary basis, with no personal gain involved.

Attendance at the Sydney conference is mandatory for those awarded travel grants. They must present their paper in person, after which the travel grant will be disbursed following their attendance.

#29

Hi, thanks for reply, it means that if the top team wants to get the prize money, they must submit a short paper and register for the conference and publish the fees. I think the prize money of the top team may not cover the conference registration fees?
①Could you list how much the conference registration fees and publishing fees will cost in total?
②The top team’s paper will be treated as a workshop paper or a main conference paper?
③At before, I didn’t have to pay the conference registration fee when I participated in the NIPS competition because of the competition was just a workshop. And this competition must also be a workshop, I think winner don’t have to register for the conference to get the prize money. If winner don’t want pulic paper and no want to pay for registration fees, can they get prize?

Wait for your reply, thanks!

#30

hi @snehananavati, can I clarify why was team freeze recently just undone for a few days and frozen again today?

#31

We were informed of an issue regarding a participant’s inability to add a team member. Upon review of the logs, it was identified that the request was made before the team freeze, but a technical glitch prevented it from being implemented.

To address the situation and ensure compliance with our processes, we made the necessary adjustments to resolve the issue while maintaining the integrity of the challenge

#32

thanks for the clarification!

#33

Hello @sergio_a

  1. What is that modified version of the Brick Schema?

We combined some of labels together. For example, various different Alarms are combined into a single class.

  1. What are the 94 point subclasses? Or how does that correspond to the 240 classes of the train_y csv? Are classes with a child not counted into these 94 classes (i.e. the 94 classes are lowest level)?

For this competition, please use the files provided from the offical competition page available at: AIcrowd . The provided train_y_v0.1.0.csv file have 94 point subclasses / “columns”.

  1. If all the datapoints are subclasses of Point and the parent classes are labeled as True. Why is the Point class in train_y -1 for all samples? Shouldn’t it be 1 because it is the parent class of everything?

In the provided files does not have the Point label / “column”.

#34

I see that the code submission stage requires providing a git repository and ensuring that you have access rights. Can you provide your git account? In this way, a private repository can be set up first and open to you. Before the end of the competition, the repository will be open to the organizer instead of being open to everyone. The purpose of this is to ensure fairness in the competition and prevent other players from seeing other team codes.

#35

Hi mike_q.

I am Arian, one of the organisers of this challenge. Here are our answers to your questions.

①Could you list how much the conference registration fees and publishing fees will cost in total?

The details for the conference are not available yet. Based on last year’s information (International World Wide Web Conference 2024 ( WWW2024 | The Web Conf 2024 ) | Registration), the early bird, non-ACM member, Workshop-only registration price is AUD 997 (SGD 850). There are no additional publication fee. The travel grants for the first to third prize will more than cover this cost.

②The top team’s paper will be treated as a workshop paper or a main conference paper?

The top teams’ papers will be treated as a workshop paper.

③At before, I didn’t have to pay the conference registration fee when I participated in the NIPS competition because of the competition was just a workshop. And this competition must also be a workshop, I think winner don’t have to register for the conference to get the prize money. If winner don’t want pulic paper and no want to pay for registration fees, can they get prize?

The winners will win the cash prize and additional travel grant. For the first place, the amounts are 5,000 AUD plus 2,500 AUD respectively, totalling 7,500 AUD.

To be eligible for the cash prize, the winners must win the competition on the combined public and secret testing set, and follow all the rules including making the documentations and code publicly available under the Apache 2.0 license.

To be eligible for the full travel grant, the winners must attend and present in the workshop in person. If that is not possible, the participants could request an exception for remote presentation directly to the conference chair. In that case, partial amount of the travel grant may be granted to cover the workshop registration cost, on a case-by-case basis.

I hope these answer your questions. I’m excited to see you and your solutions.