Someone recommended me to try out Generative Adversarial Networks, CycleGAN, MUNIT, FUNIT, pix2pix and GauGAN for instance. Deeplearning.ai has a course on GANs on Coursera, (which, I believe can be audited free,) with example notebooks covering the above algorithms. I think it would be better to share with others who are working on this problem.
2 Likes
He also told me that if the data is unpaired, then pix2pix cannot be used, and CycleGAN and MUNIT are recommended. 
Hi, the dataset is actually paired 
So you can use pix2pix!
Thanks 
But honestly, I was just trying not to steal the fun of finding the algorithm to use from participants