Open Access   Article Go Back

Comparison of various Activation Functions: A Deep Learning Approach

Mohammed Ibrahim Khan1 , Akansha Singh2 , Anand Handa3

  1. Computer Science and Engineering, PSIT College of Engineering, Kanpur, India.
  2. Computer Science and Engineering, PSIT College of Engineering, Kanpur, India.
  3. Computer Science and Engineering, PSIT College of Engineering, Kanpur, India.

Section:Research Paper, Product Type: Journal Paper
Volume-6 , Issue-3 , Page no. 122-126, Mar-2018

CrossRef-DOI:   https://doi.org/10.26438/ijcse/v6i3.122126

Online published on Mar 30, 2018

Copyright © Mohammed Ibrahim Khan, Akansha Singh, Anand Handa . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

View this paper at   Google Scholar | DPI Digital Library

XML View
PDF Download

How to Cite this Paper

  • IEEE Citation
  • MLA Citation
  • APA Citation
  • BibTex Citation
  • RIS Citation

IEEE Style Citation: Mohammed Ibrahim Khan, Akansha Singh, Anand Handa, “Comparison of various Activation Functions: A Deep Learning Approach,” International Journal of Computer Sciences and Engineering, Vol.6, Issue.3, pp.122-126, 2018.

MLA Style Citation: Mohammed Ibrahim Khan, Akansha Singh, Anand Handa "Comparison of various Activation Functions: A Deep Learning Approach." International Journal of Computer Sciences and Engineering 6.3 (2018): 122-126.

APA Style Citation: Mohammed Ibrahim Khan, Akansha Singh, Anand Handa, (2018). Comparison of various Activation Functions: A Deep Learning Approach. International Journal of Computer Sciences and Engineering, 6(3), 122-126.

BibTex Style Citation:
@article{Khan_2018,
author = {Mohammed Ibrahim Khan, Akansha Singh, Anand Handa},
title = {Comparison of various Activation Functions: A Deep Learning Approach},
journal = {International Journal of Computer Sciences and Engineering},
issue_date = {3 2018},
volume = {6},
Issue = {3},
month = {3},
year = {2018},
issn = {2347-2693},
pages = {122-126},
url = {https://www.ijcseonline.org/full_paper_view.php?paper_id=1770},
doi = {https://doi.org/10.26438/ijcse/v6i3.122126}
publisher = {IJCSE, Indore, INDIA},
}

RIS Style Citation:
TY - JOUR
DO = {https://doi.org/10.26438/ijcse/v6i3.122126}
UR - https://www.ijcseonline.org/full_paper_view.php?paper_id=1770
TI - Comparison of various Activation Functions: A Deep Learning Approach
T2 - International Journal of Computer Sciences and Engineering
AU - Mohammed Ibrahim Khan, Akansha Singh, Anand Handa
PY - 2018
DA - 2018/03/30
PB - IJCSE, Indore, INDIA
SP - 122-126
IS - 3
VL - 6
SN - 2347-2693
ER -

VIEWS PDF XML
1030 562 downloads 341 downloads
  
  
           

Abstract

A branch of machine learning that attempts to model high-level abstractions in data through algorithms by the use of multiple processing layers with complex structures and nonlinear transformations is known as Deep Learning. In this paper, we present the results of testing neural networks architectures through tensorflow for various activation functions of machine learning algorithms. It was demonstrated on MNIST database of handwritten digits in single-threaded mode that blind selection of these parameters can hugely increase the runtime without the significant increase of precision. Here, we try out different activation functions in a Convolutional Neural Network on the MNIST database and provide as results the change in loss values during training and the final prediction accuracy for all of the functions used. These results create an impactful analysis for optimization and training loss reduction strategy in image recognition problems and provide useful conclusions regarding the use of these activation functions.

Key-Words / Index Term

CNN (Convolution Neural Network), activation functions and MNIST(Modified National Institute of Standards and Technology) dataset

References

[1] Srinivas Jagirdar, K. Venkata Subba Reddy, Dr. Ahmed Abdul Moiz Qyser, “Cloud Powered Deep Learning-Emerging Trends”, International Journal of Computer Sciences and Engineering (IJCSE), Vol-4, Issue-6, 2016
[2] N. Gordienko, S. Stirenko, Yu. Kochura, O. Alienin, M. Novotarskiy, Yu. Gordienko, A. Rojbi (2017), “Deep Learning for Fatigue Estimation on the Basis of Multimodal Human-Machine Interactions”, XXIX IUPAP Conference on Computational Physics (CCP2017) (Paris, France).
[3] S. Hamotskyi, A. Rojbi, S. Stirenko, and Yu. Gordienk(2017), “Automatize Generation of Alphabets of Symbols for Multimodal HumanComputer Interfaces”, Proc. of Federated Conference on Computer Science and Information Systems, Prague (FedCSIS-2017) (Prague,Czech Republic).
[4] Zhang, Cha and Ma, Yunqian. “EnsembleMachine Learning” volume 1. Springer, 2012.
[5] Zhou, Zhi-Hua, Wu, Jianxin, and Tang,Wei. “Ensembling neural networks: many could be better than all.” Artificial Intelligence, 137(1-2):239–263, 2002.
[6] Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber, J¨urgen. “Training very deep networks”. In Advances in Neural Information Processing Systems, pp. 2377–2385, 2015.
[7] Gulcehre, Caglar, Moczulski, Marcin, Denil, Misha, and Bengio, Yoshua. “Noisy activation functions”. In International Conference on Machine Learning, pp. 3059–3068,2016a.