Getting Started Code for ECTRSY Challenge on AIcrowd¶
Author : Sainath Prasanna¶
Download Necessary Packages¶
In [ ]:
import sys
!pip install numpy
!pip install pandas
!pip install scikit-learn
Download data¶
The first step is to download out train test data. We will be training a model on the train data and make predictions on test data. We submit our predictions
In [ ]:
#Donwload the datasets
!rm -rf data
!mkdir data
!wget https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/ectrsy/v0.1/train.csv
!wget https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/ectrsy/v0.1/test.csv
!mv train.csv data/train.csv
!mv test.csv data/test.csv
Import packages¶
In [ ]:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.neural_network import MLPClassifier
from sklearn.svm import SVC
from sklearn import metrics
In [ ]:
all_data_path = "data/train.csv" #path where data is stored
In [ ]:
all_data = pd.read_csv(all_data_path) #load data in dataframe using pandas
Visualize the data¶
In [ ]:
all_data.head()
We can see the dataset contains 16 columns,where columns 1-15 shows financial data and the last column shows the 1 Month CD Rate.
Split Data into Train and Validation¶
- The next step is to think of a way to test how well our model is performing. we cannot use the test data given as it does not contain the data labels for us to verify.
- The workaround this is to split the given training data into training and validation. Typically validation sets give us an idea of how our model will perform on unforeseen data. it is like holding back a chunk of data while training our model and then using it to for the purpose of testing. it is a standard way to fine-tune hyperparameters in a model.
- There are multiple ways to split a dataset into validation and training sets. Following are two popular ways to go about it k-fold,leave one out 🧐
- Validation sets are also used to avoid your model from overfitting on the train dataset.
In [ ]:
X_train, X_val= train_test_split(all_data, test_size=0.2, random_state=42)
- We have decided to split the data with 20 % as validation and 80 % as training.
- To learn more about the train_test_split function click here. 🧐
- This is of course the simplest way to validate your model by simply taking a random chunk of the train set and setting it aside solely for the purpose of testing our train model on unseen data. as mentioned in the previous block, you can experiment 🔬 with and choose more sophisticated techniques and make your model better.
- Now, since we have our data splitted into train and validation sets, we need to get the corresponding labels separated from the data.
- with this step we are all set move to the next step with a prepared dataset.
In [ ]:
X_train,y_train = X_train.iloc[:,:-1],X_train.iloc[:,-1]
X_val,y_val = X_val.iloc[:,:-1],X_val.iloc[:,-1]
TRAINING PHASE 🏋️¶
Define the Model¶
- We have fixed our data and now we are ready to train our model.
- There are a ton of regressors to choose from like LinearRegression, etc.🧐
- Remember that there are no hard-laid rules here. you can mix and match regressors, it is advisable to read up on the numerous techniques and choose the best fit for your solution , experimentation is the key.
- A good model does not depend solely on the regressor but also on the features you choose. So make sure to analyse and understand your data well and move forward with a clear view of the problem at hand. you can gain important insight from here.🧐
In [ ]:
regressor = LinearRegression()
- To start you off, We have used a basic Linear Regression here.
- But you can tune parameters and increase the performance. To see the list of parameters visit here.
- Do keep in mind there exist sophisticated techniques for everything, the key as quoted earlier is to search them and experiment to fit your implementation.
Train the Model¶
In [ ]:
regressor.fit(X_train, y_train)
Validation Phase 🤔¶
Wonder how well your model learned! Lets check it.
Predict on Validation¶
Now we predict using our trained model on the validation set we created and evaluate our model on unforeseen data.
In [ ]:
y_pred = regressor.predict(X_val)
Evaluate the Performance¶
- We have used basic metrics to quantify the performance of our model.
- This is a crucial step, you should reason out the metrics and take hints to improve aspects of your model.
- Do read up on the meaning and use of different metrics. there exist more metrics and measures, you should learn to use them correctly with respect to the solution,dataset and other factors.
- Mean Squared Error score and Mean Absolute Error are the metrics for this challenge
In [ ]:
print('Mean Absolute Error:', metrics.mean_absolute_error(y_val, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_val, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_val, y_pred)))
Testing Phase 😅¶
We are almost done. We trained and validated on the training data. Now its the time to predict on test set and make a submission.
Load Test Set¶
Load the test data on which final submission is to be made.
In [ ]:
final_test_path = "data/test.csv"
final_test = pd.read_csv(final_test_path)
Predict Test Set¶
Predict on the test set and you are all set to make the submission !
In [ ]:
submission = regressor.predict(final_test)
Save the prediction to csv¶
In [ ]:
submission = pd.DataFrame(submission)
submission.to_csv('submission.csv',header=['1MonthCDRate'],index=False)
🚧 Note:
- Do take a look at the submission format.
- The submission file should contain a header.
- Follow all submission guidelines strictly to avoid inconvenience.
To download the generated csv in colab run the below command¶
In [ ]:
try:
from google.colab import files
files.download('submission.csv')
except:
print('not in colab')
Well Done! 👍 We are all set to make a submission and see you name on leaderborad. Let navigate to challenge page and make one.¶
In [ ]: