Metric Formula Question

CYD Campus Aircraft Localization Competition
#1

Dear organizers,

For validation purpose we tried to reproduce the evaluation metric but without success (various ways with and without haversine function). Our result does not seem to be equivalent to the leader board numbers.

Could you provide it as formula as kaggle competitions usually do?

Thank you in advance!

2 Likes
#2

Here’s the code we created:

import numpy as np
from math import radians

def RMSE_2D(X,X_pred):

    def vectorize_radians(x):
        return np.vectorize(radians)(x)

    X = vectorize_radians(X)
    X_pred = vectorize_radians(X_pred)

    R = 6373000

    lat1 = X[:,0]
    lat2 = X_pred[:,0]
    dlat = X[:,0]-X_pred[:,0]
    dlon = X[:,1]-X_pred[:,1]

    a = np.power(np.sin(dlat / 2), 2) + np.cos(lat1) * np.cos(lat2) * np.power(np.sin(dlon / 2), 2)
    c = 2 * np.arctan2(np.sqrt(a),np.sqrt(1-a))

    vector_errors = R * c

    sum_errors = np.sum(np.power(vector_errors, 2))
    return np.sqrt(sum_errors/len(vector_errors))
#3

cc: @masorx for clarification

#4

Hi,

I am not sure what you mean that you don’t get the same results as on the leaderboard (since the test data is obviously not available). I guess you are off by an order of magnitude or ore on the training data?

Haversine / Great circle distance is calculated pretty much like described here:

This distance is run on the submitted predictions vs. ground truth (NaNs are ignored, as long as 50% or more are submitted). From this, we sort and only take the best 90% of errors, hence creating a truncated RMSE and filtering possibly the worst outliers.

#5

Hi @masorx: I did not completely understand the evaluation metric.
There is a ground truth lat, lon, geoAltitude and and predicted lat, lon, geoAltitude

Let’s take lat, lon first.
Let’s say RMSE here is the 2d distance(like the medium post) between ground truth lat, lon and predicted lat, lon, for each data point and then average it all? Is it right?

How is the geoAltitude brought into picture here? RMSE between ground truth geoAltitude and predicted geoAltitude is added to the 2D distance or how it works?

For better understanding please provide a function of the evaluation metric even in R or python since I’m confused how the 2D distance is synced to the geoAltitude and get one metric value.

2 Likes