Hi everyone, please make sure that your submissions are creating prediction file with correct row_id. The row_id was not being match strictly till the previous evaluator version and we have added assert for the same now. Due to which the submissions have failed with the row_ids in the generated prediction file do not match that of the ground_truth.
Your solution need to output row_id as shared in the test data and not hardcoded / sequential (0,1,2…). Also note, that row_id can be different on data present on evaluations v/s workspace, to make sure people aren’t hardcoding from that file.
We are trying to apply automatic patch wherever possible, but it need to be ultimately fixed in solutions submitted.
Do you mean to say that row_id of 5 (example) in the given test data can be different row_id in evaluation test data?Just trying to understand how is the evaluation process - Is it on the pickle model or anything different?
"2019-12-23T07:07:16.767598332Z Resource [93mstopwords[0m not found.
2019-12-23T07:07:16.767601933Z Please use the NLTK Downloader to obtain the resource:
"
In local server,we downloaded and used it from desktop path.How do we resolve this on evaluation cluster?
Yes, the row_id i.e. 5,6,7 in test data provided to you on workspace can be anything say 1000123, 1001010, 100001 (and in random order) in test data present on the server going forward, so we know predictions are being carried out during evaluation.
To use nltk for the evaluation, you need to provide ntlk_data folder in your repository root, which can be done as follows (current working directory: at your repository root):
One more clarification is that even though row_id of test data may differ in evaluation cluster,hope the other content is intact as made available shared path testing_data_full.
This demonstrates what is happening in your submission along with alternatives which you can use (name of variable changed to hide any potential information getting public on feature used):
(suggested ways #1, decently memory efficient)
>>> something_stack_2 = pd.get_dummies(something_stack)
>>> something_stack_2.info()
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 38981 entries, (0, 0) to (8690, 2)
Columns: 4589 entries, to <removed>
dtypes: uint8(4589)
memory usage: 170.7 MB
(suggested ways #2, most memory efficient, slower then #1)
>>> something_stack_2 = pd.get_dummies(something_stack, sparse=True)
>>> something_stack_2.info()
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 38981 entries, (0, 0) to (8690, 2)
Columns: 4589 entries, to <removed>
dtypes: Sparse[uint8, 0](4589)
memory usage: 304.8 KB
(what your submission is doing -- ~5G was available at this time)
>>> something_stack_2 = something_stack.str.get_dummies()
Killed
NOTE: The only difference between two approaches is Series.str.get_dummies use “|” as separator by default. In case you were relying on it, can do something like below:
Let us know in case the problem continues after changing this (here and it’s usage anywhere else in your codebase), we will be happy to debug further accordingly.
Is there any way for the users to see the error cause by themselves?It feels odd to post here to know the reason of error every single time without any clue on trace back.