Update in mono-depth evaluation method

Mono Depth Perception
#1

Hi everyone,

We have made a change in the evaluation setup, to allow for models that predict relative depth maps from mono image inputs. We now apply an alignment step to the model to align a relative depth prediction to the ground truth. For more details please refer to this paper Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer.

Updated code for local evaluation of your models can be found in the starter-kit . All valid old submissions will be re-evaluated. All future submissions will be evaluated with these updated metrics.

#2

Something wrong in updated mono-depth evaluation method. I had submited my np.random.choice predictor and got the 1-st in LB.

3 Likes
#3

Hi Victor,

Thanks for pointing this out, the SILog had gone to NaN because your predictions were too large, I think you probably output the depth values without the scaling factor as shown in local_evaluation.py? The submission is now marked as failed.

Please check this function for reading depth images. The values should be closer to ~1-50.

#4

@dipam,
I have a similar issue with my submission: I don’t believe my Silog is equal to 0 and I m estimating in the 8 to 10 range. all the other metrics similar to what I have locally.

#5

@xav_c

I did notice that, however your submission didn’t get NaN scores. I’ll need some time to investigate, should be resolved by tomorrow.

1 Like
#6

@dipam, I tried twice to submit new weights with identical code to my last “evaluation successful” submission (everything the same but the weights) submitted a week ago, I am getting an error: ““Evaluate Scores: AssertionError:”” and no additional information. the logs are identical by the way.
Could you please tell me more about the error as I am out of ideas…