Our main solution consisted of three parts: A large pre-trained vision transformer model (microsoft/beit-large-patch16-512 · Hugging Face), a modified version of the baseline SimCLR model, and a large number of hand-crafted features (using the keypoints). These were combined by weighted PCA, where the weight was both column-wise (with different weights given to the three parts above), and row-wise (with more weight given to frames with a lot of movement).
We also tried different ways of encoding the time series of keypoints, in particular different BERT-inspired methods, but also ROCKET ([1910.13051] ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels). However, the results we obtained were not good enough to include in the main solution.