I am writing a document which compile more than 30 activation functions, and I am wondering if I can classify these functions into some main families.
I am mostly self taught and I probably lack some academic background to really & deeply understand what I am doing, so something that might be obvious to you might not be obvious to me.
After some time studying the various activation functions I gathered in books or online, I concluded that I could probably classify them into the following types :
- Unipolar Binary
- Bipolar Binary
- Unipolar Continuous
- Bipolar Continuous
- Linear
But some documentation I read confuse me and I am not sure any-more that I can use these terms to qualify multiple functions.
For instance, if I produce the following classification, am I right?
Unipolar Binary
- Unit Step (Step)
Bipolar Binary
- Sig (Signum / Binary Step)
Unipolar Continuous
- Sigmoid (Logistic)
- Complementary log-log
- ReLU
- Smooth ReLU (Smooth Rectifier / Smooth max / Soft plus)
- Gaussian
Bipolar Continuous
- Bipolar Sigmoid
- Tanh
- Hard Tanh *(--> Linear ?)
- ArcTan
- LeCun's Tanh
- Softsign
- Absolute -(--> Linear ?)
- LReLU (--> Linear ?)
- PReLU (--> Linear ?)
- RReLU (--> Linear ?)
- ELU
- SELU
- SReLU (--> Linear ?)
- Piece-wise Linear (--> Linear ?)
- Adaptative Piece-wise Linear (--> Linear ?)
- Bend Indentity
- Soft Exponential
- Sinusoid
- Sinc
- Logit
- Probit
- Cosine
- Softmax
- Maxout (--> Linear ?)
Linear
- Identity
Beside the very probable classification errors, does this classification makes sense or am I totally wrong?
Also, as you can see in this sample, I found what seems to be similar functions under several names (e.g.: Sig / Signum / Binary Step). If this is really the same functions, why is there so many names for a given function?