Baseline - ORIENTME

ashivani · May 2, 2020, 10:28am

Baseline for ORIENTME Challenge on AIcrowd¶

Author : Sharada Mohanty¶

To open this notebook on Google Colab, click below!¶

Download Necessary Packages¶

In [0]:

import sys
!{sys.executable} -m pip install numpy
!{sys.executable} -m pip install pandas
!{sys.executable} -m pip install scikit-learn 
!{sys.executable} -m pip install matplotlib tqdm

Download data¶

The first step is to download the training data and the test data

In [ ]:

# #Donwload the datasets
!rm -rf data/
!mkdir data/

!curl https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/orientme/v0.2/training.tar.gz -o data/training.tar.gz
!curl https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/orientme/v0.2/test.tar.gz -o data/test.tar.gz
!curl https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/orientme/v0.2/sample_submission.csv -o data/sample_submission.csv
!tar xvzf data/training.tar.gz -C data/
!tar xvzf data/test.tar.gz -C data/

In [23]:

## Now the data is available at the following locations:

TRAINING_IMAGES_FOLDER = "data/training/images/"
TRAINING_LABELS_PATH = "data/training/labels.csv"
TEST_IMAGES_FOLDER = "data/images"
SAMPLE_SUBMISSION_FILE_PATH = "data/sample_submission.csv"

Import packages¶

In [24]:

import os
import tqdm

import pandas as pd
import numpy as np

from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error,mean_absolute_error
import matplotlib.pyplot as plt
%matplotlib inline

from PIL import Image

Load Data¶

We use PIL library to load our images. Here we are creating our array where our input features are the mean colours and output features are the rotations along the 3 axis.

In [25]:

training_labels_df = pd.read_csv(TRAINING_LABELS_PATH)

def pre_process_data_X(image):
    """
    This file takes a loaded image and returns a particular 
    representation of the data point
    
    
    NOTE: This current baseline implements a **very** silly approach
    of representing every image by the mean RGB values for every image.
    
    You are encourage to try to alternate representations of the data,
    or figure out how to learn the best representation from the data ;)
    """
    im_array = np.array(im)
    mean_rgb = im_array.mean(axis=(0, 1))
    return mean_rgb


ALL_DATA = []

for _idx, row in tqdm.tqdm(training_labels_df.iterrows(), total=training_labels_df.shape[0]):
    filepath = os.path.join(
        TRAINING_IMAGES_FOLDER,
        row.filename
    )
    im = Image.open(filepath)
    
    data_X = pre_process_data_X(im)
    data_Y = [row.xRot]
    
    ALL_DATA.append((data_X, data_Y))

100%|██████████| 5000/5000 [00:54<00:00, 91.61it/s]

Exploratory Data Analysis¶

We now see the kind of images the dataset contains to get a better idea. The title signifies clockwise rotation of the cube along that axis

In [26]:

plt.figure(figsize=(20,20))
for i in range(16):
  filename,xRot = training_labels_df.iloc[i]
  filepath = os.path.join(
        TRAINING_IMAGES_FOLDER,
        filename
    )
  im = Image.open(filepath)
  plt.subplot(4,4,i+1)
  plt.axis('off')
  plt.title("xRot: %.3f"%(xRot))
  plt.imshow(im)

Split Data into Train and Validation¶

We split the dataset into Training data and Validation datasets to help us test the generalizability of our models, and to ensure that we are not overfitting on the training set.

In [27]:

training_set, validation_set= train_test_split(ALL_DATA, test_size=0.2, random_state=42)

Here we have selected the size of the testing data to be 20% of the total data. You can change it and see what effect it has on the accuracies. To learn more about the train_test_split function click here.

Now, since we have our data splitted into train and validation sets, we need to get the label separated from the data.

In [28]:

X_train, y_train = zip(*training_set)
X_val, y_val = zip(*validation_set)


X_train = np.array(X_train)
y_train = np.array(y_train)
X_val = np.array(X_val)
y_val = np.array(y_val)

Define the Classifier¶

Now we finally come to the juicy part. Now that all the data is all loaded and available nice, we can finally get to training the classifier. Here we use sklearn MLPRegressor to train our network. We can tune the hyper parameters based on cross validation scores

In [29]:

model = MLPRegressor(hidden_layer_sizes=[10, 10], verbose=True)
# NOTE : This is again silly hyper parameter instantiation of this problem,
# and we encourage you to explore what works the best for you.

Train the classifier¶

In [30]:

model.fit(X_train, y_train)

/Users/mohanty/miniconda3/lib/python3.7/site-packages/sklearn/neural_network/_multilayer_perceptron.py:1342: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)

Iteration 1, loss = 21026.42211007
Iteration 2, loss = 19387.25011615
Iteration 3, loss = 17776.03423797
Iteration 4, loss = 16206.76782771
Iteration 5, loss = 14600.49613243
Iteration 6, loss = 12960.55356979
Iteration 7, loss = 11324.92016918
Iteration 8, loss = 9863.40099234
Iteration 9, loss = 8659.87262911
Iteration 10, loss = 7713.43451266
Iteration 11, loss = 7078.02702196
Iteration 12, loss = 6675.90558761
Iteration 13, loss = 6463.65494875
Iteration 14, loss = 6320.13137236
Iteration 15, loss = 6215.66516995
Iteration 16, loss = 6114.79442940
Iteration 17, loss = 6016.75085442
Iteration 18, loss = 5919.08684030
Iteration 19, loss = 5816.70833979
Iteration 20, loss = 5717.03973663
Iteration 21, loss = 5614.25882606
Iteration 22, loss = 5513.11119881
Iteration 23, loss = 5413.86707581
Iteration 24, loss = 5315.36311634
Iteration 25, loss = 5218.94132154
Iteration 26, loss = 5126.89914806
Iteration 27, loss = 5039.04441691
Iteration 28, loss = 4962.25329710
Iteration 29, loss = 4877.70589864
Iteration 30, loss = 4807.57701642
Iteration 31, loss = 4744.36290424
Iteration 32, loss = 4687.76792486
Iteration 33, loss = 4634.70041976
Iteration 34, loss = 4588.63638178
Iteration 35, loss = 4550.57774566
Iteration 36, loss = 4514.44416680
Iteration 37, loss = 4486.29455697
Iteration 38, loss = 4464.49088720
Iteration 39, loss = 4445.10933753
Iteration 40, loss = 4430.03473835
Iteration 41, loss = 4414.50496911
Iteration 42, loss = 4400.76699672
Iteration 43, loss = 4393.47247598
Iteration 44, loss = 4384.29465022
Iteration 45, loss = 4379.09032611
Iteration 46, loss = 4376.96583051
Iteration 47, loss = 4369.42780817
Iteration 48, loss = 4373.09690940
Iteration 49, loss = 4361.63563369
Iteration 50, loss = 4356.83169231
Iteration 51, loss = 4358.34106896
Iteration 52, loss = 4352.57402505
Iteration 53, loss = 4349.91589264
Iteration 54, loss = 4352.05740412
Iteration 55, loss = 4343.69820814
Iteration 56, loss = 4344.72570902
Iteration 57, loss = 4343.26483857
Iteration 58, loss = 4338.94459592
Iteration 59, loss = 4336.32408098
Iteration 60, loss = 4335.72070751
Iteration 61, loss = 4330.14549346
Iteration 62, loss = 4332.17681920
Iteration 63, loss = 4325.72766839
Iteration 64, loss = 4328.30871668
Iteration 65, loss = 4321.93376744
Iteration 66, loss = 4320.91176755
Iteration 67, loss = 4318.88632987
Iteration 68, loss = 4313.42089144
Iteration 69, loss = 4314.83191256
Iteration 70, loss = 4308.86172168
Iteration 71, loss = 4306.97792465
Iteration 72, loss = 4304.33177687
Iteration 73, loss = 4305.90349935
Iteration 74, loss = 4299.04114350
Iteration 75, loss = 4296.49780976
Iteration 76, loss = 4291.70147767
Iteration 77, loss = 4290.30773836
Iteration 78, loss = 4286.67534210
Iteration 79, loss = 4284.28657554
Iteration 80, loss = 4280.71821816
Iteration 81, loss = 4275.34265437
Iteration 82, loss = 4276.23303646
Iteration 83, loss = 4269.23646740
Iteration 84, loss = 4266.22411026
Iteration 85, loss = 4266.28870615
Iteration 86, loss = 4260.66975049
Iteration 87, loss = 4257.07082207
Iteration 88, loss = 4251.88791110
Iteration 89, loss = 4247.42304561
Iteration 90, loss = 4241.15104897
Iteration 91, loss = 4241.21460406
Iteration 92, loss = 4233.82885298
Iteration 93, loss = 4231.13886906
Iteration 94, loss = 4226.70853004
Iteration 95, loss = 4226.64258036
Iteration 96, loss = 4219.01710227
Iteration 97, loss = 4215.32342634
Iteration 98, loss = 4212.53672884
Iteration 99, loss = 4207.75181728
Iteration 100, loss = 4203.63557113
Iteration 101, loss = 4199.66034317
Iteration 102, loss = 4201.06173166
Iteration 103, loss = 4194.76290233
Iteration 104, loss = 4189.33164207
Iteration 105, loss = 4184.27395827
Iteration 106, loss = 4183.47306163
Iteration 107, loss = 4180.86284377
Iteration 108, loss = 4175.98295856
Iteration 109, loss = 4171.36620125
Iteration 110, loss = 4167.08502369
Iteration 111, loss = 4157.55291436
Iteration 112, loss = 4154.99399921
Iteration 113, loss = 4149.45642220
Iteration 114, loss = 4144.27572560
Iteration 115, loss = 4141.19416195
Iteration 116, loss = 4136.80386588
Iteration 117, loss = 4131.61698724
Iteration 118, loss = 4127.09033853
Iteration 119, loss = 4122.18997896
Iteration 120, loss = 4121.30944343
Iteration 121, loss = 4117.99987951
Iteration 122, loss = 4109.50285926
Iteration 123, loss = 4106.16572266
Iteration 124, loss = 4100.22470345
Iteration 125, loss = 4094.65405184
Iteration 126, loss = 4088.54033152
Iteration 127, loss = 4086.22685436
Iteration 128, loss = 4081.24372484
Iteration 129, loss = 4078.66309559
Iteration 130, loss = 4071.81210450
Iteration 131, loss = 4062.52670483
Iteration 132, loss = 4059.09073378
Iteration 133, loss = 4052.26907766
Iteration 134, loss = 4046.96719303
Iteration 135, loss = 4042.80876239
Iteration 136, loss = 4036.87714743
Iteration 137, loss = 4029.78945219
Iteration 138, loss = 4023.84547338
Iteration 139, loss = 4018.33271727
Iteration 140, loss = 4013.03512785
Iteration 141, loss = 4008.30768027
Iteration 142, loss = 4003.51395810
Iteration 143, loss = 3994.39388498
Iteration 144, loss = 3988.39967183
Iteration 145, loss = 3995.71194494
Iteration 146, loss = 3976.31718163
Iteration 147, loss = 3973.52908405
Iteration 148, loss = 3970.25365657
Iteration 149, loss = 3959.02029431
Iteration 150, loss = 3952.72404385
Iteration 151, loss = 3947.53475801
Iteration 152, loss = 3937.89993309
Iteration 153, loss = 3932.57151416
Iteration 154, loss = 3926.37400201
Iteration 155, loss = 3921.15662014
Iteration 156, loss = 3909.37692989
Iteration 157, loss = 3908.96759180
Iteration 158, loss = 3896.38136475
Iteration 159, loss = 3888.24491819
Iteration 160, loss = 3883.67117099
Iteration 161, loss = 3880.66782721
Iteration 162, loss = 3889.24411119
Iteration 163, loss = 3879.03957322
Iteration 164, loss = 3862.27590234
Iteration 165, loss = 3849.96038462
Iteration 166, loss = 3844.52707828
Iteration 167, loss = 3837.86534712
Iteration 168, loss = 3823.52771532
Iteration 169, loss = 3820.28701587
Iteration 170, loss = 3806.60542535
Iteration 171, loss = 3800.31685360
Iteration 172, loss = 3790.81910798
Iteration 173, loss = 3783.54430430
Iteration 174, loss = 3786.12668525
Iteration 175, loss = 3774.95702730
Iteration 176, loss = 3760.68112271
Iteration 177, loss = 3750.86951336
Iteration 178, loss = 3743.54787278
Iteration 179, loss = 3736.08646552
Iteration 180, loss = 3729.95243674
Iteration 181, loss = 3719.97806029
Iteration 182, loss = 3711.99382349
Iteration 183, loss = 3709.42922806
Iteration 184, loss = 3693.25850998
Iteration 185, loss = 3683.70908169
Iteration 186, loss = 3674.19976363
Iteration 187, loss = 3665.41089527
Iteration 188, loss = 3677.26170767
Iteration 189, loss = 3649.82712815
Iteration 190, loss = 3640.26249305
Iteration 191, loss = 3633.77859420
Iteration 192, loss = 3624.30667511
Iteration 193, loss = 3620.55348946
Iteration 194, loss = 3613.44074447
Iteration 195, loss = 3604.63260268
Iteration 196, loss = 3602.69793201
Iteration 197, loss = 3584.42236278
Iteration 198, loss = 3580.94173022
Iteration 199, loss = 3572.08735237
Iteration 200, loss = 3560.95677697

/Users/mohanty/miniconda3/lib/python3.7/site-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)

Out[30]:

MLPRegressor(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,
             beta_2=0.999, early_stopping=False, epsilon=1e-08,
             hidden_layer_sizes=[10, 10], learning_rate='constant',
             learning_rate_init=0.001, max_fun=15000, max_iter=200,
             momentum=0.9, n_iter_no_change=10, nesterovs_momentum=True,
             power_t=0.5, random_state=None, shuffle=True, solver='adam',
             tol=0.0001, validation_fraction=0.1, verbose=True,
             warm_start=False)

Predict on Validation¶

Now we predict our trained classifier on the validation set and evaluate our model

In [31]:

y_pred = model.predict(X_val)

Evaluate the Performance¶

We use the same metrics as that will be used for the test set.
MAE and RMSE are the metrics for this challenge

In [32]:

print('Mean Absolute Error:', mean_absolute_error(y_val, y_pred))  
print('Mean Squared Error:', mean_squared_error(y_val, y_pred))  
print('Root Mean Squared Error:', np.sqrt(mean_squared_error(y_val, y_pred)))

Mean Absolute Error: 69.53877043065042
Mean Squared Error: 7235.278760528952
Root Mean Squared Error: 85.06044180774605

Load Test Set¶

Load the test data now

In [33]:

import glob

TEST_DATA = []
TEST_FILENAMES = []

for _test_image_path in tqdm.tqdm(glob.glob(os.path.join(TEST_IMAGES_FOLDER, "*.jpg"))):
    filename = os.path.basename(_test_image_path)
    im = Image.open(_test_image_path)
    
    data_X = pre_process_data_X(im)
    TEST_DATA.append(data_X)
    TEST_FILENAMES.append(filename)

100%|██████████| 5001/5001 [00:56<00:00, 88.68it/s]

In [34]:

TEST_DATA = np.array(TEST_DATA)
# model = classifier

Make predictions on the test set¶

In [35]:

test_predictions = model.predict(TEST_DATA)

In [36]:

test_df = pd.DataFrame(test_predictions, columns=['xRot'])
test_df["filename"] = TEST_FILENAMES

Save the prediction to csv¶

In [37]:

test_df.to_csv('submission.csv', index=False)

Note: Do take a look at the submission format.The submission file should contain the following header : filename,xRot.

To download the generated csv in Google Colab run the below command¶