With the capability to self-learn and succeed to achieve favourable results in various classification problem, deep learning techniques are increasingly used for Automatic Facial Expression Recognition (AFER). In this paper, we provide brief survey on deep learning technique particularly Convolutional Neural Networks (CNN) for Facial Expression Recognition (FER) and newly introduced Infrared based FER dataset. This review is focused on various CNN techniques applied in almost last half decade for FER on Infrared and Visible light images. There are certain unique advantages of using thermal and infrared images which can make FER techniques robust. Paper describes the standard flow of deep facial expression recognition and suggested methods based on research conducted specifically in this area. Later, review of existing novel deep neural networks and implementations for still images and video-based FER for Infrared Images is provided which subsequently follows glimpses of available well-known datasets. Since all types of cameras experience price reduction over the years, in near future integration and usage of such cameras would be common also because of its illumination invariant characteristic. It becomes evident at the end of the paper that there is a definite scope of developing promising and robust FER with use of Infrared and Thermal images.

Convolution Neural Networks, Deep Learning, Facial Expression Recognition, Infrared Images

[1] P. Ekman ; W.V Friesen,. “Facial Action Coding System: Investigator’s Guide”, 1st ed.; Consulting Psychologists Press: Palo Alto, CA, USA, pp. 1–15, 1978., ISBN 9993626619.
[2] D. Matsumoto, “More evidence for the universality of a contempt expression,” Motivation and Emotion, vol.16, no.4, pp.363–368,1992.
[3] Happy, S.L.; George, A.; Routray, “ A real time facial expression classification system using local binary patterns”. In Proceedings of the 4th International Conference on Intelligent Human Computer Interaction, Kharagpur, India, pp. 1–5, 2012.
[4] G. Zhao and M. Pietikainen, “Dynamic texture recognition using local binary patterns with an application to facial expressions,” IEEE transactions on pattern analysis and machine intelligence, vol. 29, no. 6, pp. 915–928, 2007.
[5] A.S. Mali, A.A. Kenjale, P.M. Ghatage, A.G. Deshpande, "Mood based Music System", International Journal of Scientific Research in Computer Science and Engineering, Vol.6, Issue.3, pp.27-30, 2018
[6] S. Ioannou, V. Gallese, and A. Merla, “Thermal infrared imaging in psychophysiology: potentialities and limits,” Psychophysiology, vol. 51, no. 10, pp. 951–963, 2014.
[7] M. Kopaczka, K. Acar, and D. Merhof, “Robust facial landmark detection and face tracking in thermal infrared images using active appearance models,” in International Conference on Computer Vision Theory and Applications (VISAPP), Rome, Italy, pp. 150–158, February 2016
[8] S. He, S. Wang, W. Lan, H. Fu and Q. Ji, "Facial Expression Recognition Using Deep Boltzmann Machine from Thermal Infrared Images," 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, pp. 239-244, 2013.
[9] P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, pp. I-I. 2001.
[10] Mollahosseini, D. Chan, and M. H. Mahoor, “Going deeper in facial expression recognition using deep neural networks,” in Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE, pp. 1–10,2016.
[11] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for facial expression recognition,” in Computer Vision (ICCV), 2015 IEEE International Conference on. IEEE, pp. 2983–2991, 2015.
[12] V. Kazemi and J. Sullivan, "One millisecond face alignment with an ensemble of regression trees," IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, pp. 1867-1874, 2014.
[13] P. Simard, D. Steinkraus, J. C. Platt,” Best practices for convolutional neural networks applied to visual document analysis”, in Proceedings Seventh International Conference on Document Analysis and Recognition, pp. 958–963, 2003.
[14] A.T.Lopes, E.de Aguiar, A.F.DeSouza, and T.Oliveira-Santos,‘‘Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order,’’ Pattern Recognition., vol. 61, pp. 610–628, Jan. 2017.
[15] D. A. Pitaloka, A. Wulandari, T. Basaruddin, and D. Y. Liliana, “Enhancing cnn with preprocessing stage in automatic emotion recognition,” Procedia Computer Science, vol. 116, pp. 523–529, 2017.
[16] W. Li, M. Li, Z. Su, and Z. Zhu, “A deep-learning approach to facial expression recognition with candid images,” in Machine Vision Applications (MVA), 2015 14th IAPR International Conference on. IEEE, pp. 279–282,2015.
[17] Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.
[18] Maas, Andrew L.. “Rectifier Nonlinearities Improve Neural Network Acoustic Models.” (2013).
[19] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision, pp. 1026–1034, 2015.
[20] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015
[21] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580, 2012(dropouts)
[22] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[23] C. Szegedy et al., "Going deeper with convolutions," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, pp. 1-9, 2015.
[24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016
[25] O`Shea, Keiron & Nash, Ryan. “An Introduction to Convolutional Neural Networks”, ArXiv e-prints.2015
[26] Peng, M.; Wang, C.; Chen, T.” NIRFaceNet: A Convolutional Neural Network for Near-Infrared Face Identification.” Information vol. 7, no. 4:61, 2016
[27] Corneanu, C.A; Simon, M.O.; Cohn, J.F.; Guerrero, S.E. “Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications”. IEEE Trans. Pattern Anal. Mach. Intell., 38, 1548–1568, 2016.
[28] Zhang, Wei & Zhang, Youmei & Ma, Lin & Guan, Jingwei & Gong, Shijie.” Multimodal learning for facial expression recognition “Pattern Recognition, 48, 10.1016/j.patcog.2015.04.012, (2015).
[29] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on. IEEE, pp. 94–101, 2010.
[30] M. J. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, and J. Budynek, “The japanese female facial expression (jaffe) database,” 1998.
[31] G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. Pietik¨aInen, “Facial expression recognition from near-infrared videos,” Image and Vision Computing, vol. 29, no. 9, pp. 607–619, 2011
[32] S. Wang, Z. Liu, S. Lv, Y. Lv, G. Wu, P. Peng, F. Chen, and X. Wang, “A natural visible and infrared facial expression database for expression recognition and emotion inference,” IEEE Transactions on Multimedia, vol. 12, no. 7, pp. 682–691, 2010.
[33] M. Kopaczka, R. Kolk and D. Merhof, "A fully annotated thermal face database and its application for thermal facial expression recognition" 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Houston, TX, pp. 1-6, 2018.
[34] B. Yang, J. Cao, R. Ni and Y. Zhang, "Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images," in IEEE Access, vol. 6, pp. 4630-4640, 2018.
[35] Fathallah, L. Abdi and A. Douik, "Facial Expression Recognition via Deep Learning," 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), Hammamet, pp. 745-750, 2017.
[36] B. Hasani and M. H. Mahoor, “Facial expression recognition using enhanced deep 3d convolutional neural networks,” in Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. IEEE, pp. 2278–2288, 2017
[37] Hernández B, Olague G, Hammoud R, Trujillo L, Romero E. “Visual learning of texture descriptors for facial expression recognition in thermal imagery”. Computer Vision and Image Understanding. 2007.
[38] S. Wang, M. He, Z. Gao, S. He, and Q. Ji, “Emotion recognition from thermal infrared images using deep boltzmann machine,” FCS, vol. 8, no. 4, pp. 609–618, 2014.
[39] Shen, P.; Wang, S.; Liu, Z. “Facial expression recognition from infrared thermal videos”. Intell. Auton. Syst, 12, 323–333, 2013.
[40] Wang, S., He, S., Wu, Y. et al. “Fusion of visible and thermal images for facial expression recognition” Front. Comput. Sci. 8: 232-242, 2014.
[41] Z. Wu, T. Chen, Y. Chen, Z. Zhang, and G. Liu, “Nirexpnet: Three stream 3d convolutional neural network for near infrared facial expression recognition,” Applied Sciences, vol. 7, no. 11, pp. 1184-1197, 2017
[42] L. Trujillo, G. Olague, R. Hammoud, and B. Hernandez, “Automatic feature localization in thermal images for facial expression recognition,” in Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 14–14, 2005.
[43] G.Sowmiya, V. Kumutha, "Facial Expression Recognition Using Static Facial Images", International Journal of Scientific Research in Computer Science and Engineering, Vol.6, Issue.2, pp.72-75, 2018