Submission failed and there's no informative stack trace

My submission failed, but I don’t quite know why. Here are the logs:

===========================
Partial run on init: None
All music names on init: ['SS_018', 'SS_AM_Contra', 'SS_008']
Overriding time out for SS_AM_Contra to 320 seconds
inference took:  69.58658599853516
inference took:  68.44680309295654
inference took:  68.7212233543396
inference took:  68.0718355178833
/dataset/SS_AM_Contra/mixture.wav: prediction completed.
Overriding time out for SS_008 to 167 seconds
inference took:  36.14090633392334
inference took:  35.45663785934448
inference took:  36.21856451034546
inference took:  36.21227216720581
/dataset/SS_008/mixture.wav: prediction completed.
Overriding time out for SS_018 to 272 seconds
inference took:  57.868829011917114
inference took:  57.00443768501282
inference took:  56.74909567832947
inference took:  57.35631251335144
/dataset/SS_018/mixture.wav: prediction completed.
Successfully generated predictions!
2021-06-28 15:56:21.473107: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-28 15:56:21.474925: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-06-28 15:57:29.938660: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-28 15:57:30.118396: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2499995000 Hz

Are there any other more informative logs? This is the issue: https://gitlab.aicrowd.com/ada_or_ardor/music-demixing-challenge-starter-kit/-/issues/5

Hi @ada_or_ardor, your generated files have a different shape than the mixture file due to which it is failing.

## These logs are part of admin code, due to which below error message isn't visible to you automatically

numpy arrays of shape
    [n_sources x n_samples x n_channels]

AssertionError: Shape mismatch between references ((4, 7612339, 2)) and estimates ((4, 7611392, 2)).
AssertionError: Shape mismatch between references ((4, 4660120, 2)) and estimates ((4, 4659200, 2)).
AssertionError: Shape mismatch between references ((4, 4660120, 2)) and estimates ((4, 4659200, 2)).

Please check your prediction code in case it is omitting samples or something similar.

You can check locally by using the following commands (output shared below is for Mu - Too Bright song using your code):

In [34]: bass, rate = sf.read("data/results/Mu - Too Bright/bass.wav")
In [35]: bass.shape
Out[35]: (308224, 2)

In [37]: mixture, rate = sf.read("data/test/Mu - Too Bright/mixture.wav")
In [38]: mixture.shape
Out[38]: (308700, 2)