Experiments with “unlabelled” data

Data Purchasing Challenge 2022
#1

Here are just my results. I used the same model, but different purchase modes.

  1. Train with initial 5000 images only: LB 0.869
  2. Add 3000 random images from unlabelled dataset: 0.881
  3. “smart” purchasing (at least non random): 0.888

So we see, that using some “smart” purchasing is helpful, but not so many, maybe ~0.01.
Probably tuning models would be more helpful to push further.

3 Likes
[Notebooks & Resource] Community Contributions and Baselines
#2

@sergey_zlobin : Thanks for your information.

I am wondering if you tried to purchase all 10K then what the score could be.

#3

I wrote scores from the leaderboard. I can’t check 10K there…
Local scores are a little bit higher than LB, but correlated with LB.
Yeah maybe I’ll check it locally.

#4

I’ve checked it locally.
Using all 10K images is better than my 3K choosing by 0.006. Maybe I can take some of it by changing purchasing algorithm. But still I feel I need to tune my model.

#5

yeah, looks like your model/training pipeline is limiting you. You should be able to get a much bigger improvement from using all labels. Maybe try a bigger model and tune it a bit.

#6

Do you mean the reported score by @sergey_zlobin during the pre-training phase? or with the purchase phase?

#7

I mean the addition of 10000 vs 3000 shouldn’t result in only such a modest improvement.

#8

Aah, yes. With the post. I think maybe there are only some ~4000-5000 images good enough for improving the system and therefore adding all 10000 doesn’t make much improvement because the basic model will predict them correctly anyway.
It’s just a theory, not tested. I haven’t touched the purchase part yet.
Thanks for your insight.

#9

yes, it’s very significant.

From my experiment notebook its something like this :

exp no. augmentation pretrained purchase_method score_pretraining_phase score_purchase_phase score_validation_phase LB_Score
1 NO NO NO 0.773 0.773 0.760
2 NO NO RANDOM 3000 0.773 0.804 0.760
3 NO NO ALL 10000 0.773 0.841 0.835
4 NO YES NO 0.857 0.857 0.850
5 NO YES RANDOM 3000 0.857 0.864 0.845 0.851
6 NO YES ALL 10000 0.857 0.892 0.875
7 YES YES NO 0.868 0.868 0.865
8 YES YES RANDOM 3000 0.868 0.886 0.869 0.880
9 YES YES ALL 10000 0.868 0.902 0.893

the notebook :

1 Like