Evaluation metric

CYD Campus Aircraft Localization Competition
#1

I did not see the query answered so I have created another discussion topic

I did not completely understand the evaluation metric.
There is a ground truth lat, lon, geoAltitude and and predicted lat, lon, geoAltitude

Let’s take lat, lon first.
Let’s say RMSE here is the 2d distance(like the medium post) between ground truth lat, lon and predicted lat, lon, for each data point and then average it all? Is it right?

How is the geoAltitude brought into picture here? RMSE between ground truth geoAltitude and predicted geoAltitude is added to the 2D distance or how it works?

For better understanding please provide a function of the evaluation metric even in R or python since I’m confused how the 2D distance is synced to the geoAltitude and get one metric value.

#2

cc: @masorx for clarification

#3

Hi, altitude is ignored entirely for evaluation in this round (hence 2D). We may have a look at this later.
The main reason is based on aviation history: barometric altitude is available and provided separately, and is of course highly correlated with the geometric one. It’s really just RMSE on the lat/lon values.

#4

I try to put some more details (always waiting confirmation by @masorx).
Using the definition of RMSE:

  1. compute the 2D distance between the true position and estimated position for each point, for example using the Haversine formula,
  2. Obtain a series of distances {d_i} where i=1…N;
  3. Compute sqrt(sum(d_i^2)/N);
  4. Last, there is a truncation: only the best estimations -the ones having the smaller distances- are used, i.e. dropping out the 10%.