Activation functions

• Activation functions
  • Creating a layer
    • Activation parameters holder
  • Layer types

This section describes the layers that calculate the value of specified activation functions for each of their inputs.

Creating a layer

CPtr<CBaseLayer> NEOML_API CreateActivationLayer( const CActivationDesc& activation );

Creates a layer that calculates the activation function specified by the activation parameter.

Activation parameters holder

To create a layer, one must specify its type and, optionally, parameters. All configurable activation layers have a CParam structure with all possible settings.

class CActivationDesc {
public:
	// For non-parametrized activations or default parameters
	CActivationDesc( TActivationFunction type );

	// For explicitly setting activation parameters. 'Param' must be a 'CParam' struct from the correspondent layer
	template<class Param>
	CActivationDesc( TActivationFunction type, const Param& param );

	// Changing/setting parameters of the selected activation.
	// 'Param' must be a 'CParam' struct from the correspondent layer.
	template<class Param>
	void SetParam( const Param& param );

	/// etc.
};

Layer types

enum TActivationFunction {
	AF_Linear = 0,
	AF_ELU,
	AF_ReLU,
	AF_LeakyReLU,
	AF_Abs,
	AF_Sigmoid,
	AF_Tanh,
	AF_HardTanh,
	AF_HardSigmoid,
	AF_Power,
	AF_HSwish,
	AF_GELU,
	AF_Exp,
	AF_Log,

	AF_Count
};

TActivationFunction constant	Class name	Activation function
AF_Linear	CLinearLayer	a linear activation function: ax + b
AF_ELU	CELULayer	ELU activation function
AF_ReLU	CReLULayer	ReLU activation function
AF_LeakyReLU	CLeakyReLULayer	LeakyReLU activation function
AF_Abs	CAbsLayer	abs(x) activation function
AF_Sigmoid	CSigmoidLayer	sigmoid activation function
AF_Tanh	CTanhLayer	tanh activation function
AF_HardTanh	CHardTanhLayer	HardTanh activation function
AF_HardSigmoid	CHardSigmoidLayer	HardSigmoid activation function
AF_Power	CPowerLayer	pow(x, exp) activation function
AF_HSwish	CHSwishLayer	h-swish activation function
AF_GELU	CGELULayer	x * F( X < x ) activation function, where X ~ N(0, 1)
AF_Exp	CExpLayer	exp activation function
AF_Log	CLogLayer	log activation function
AF_Erf	CErfLayer	erf activation function
AF_Count		This is an auxiliary constant: it contains the number of supported activation functions.