\@writefile{lot}{\contentsline{table}{\numberline{1.2}{\ignorespaces Recal of the most popular complex-valued activation functions.\relax}}{10}{table.caption.20}\protected@file@percent }
\newlabel{tab:cmplx_activations}{{1.2}{10}{Recal of the most popular complex-valued activation functions.\relax}{table.caption.20}{}}
@@ -199,7 +199,7 @@ This can be done easily with both the sigmoid and the hyperbolic tangent, mappin
Notice that this approach maps the phase of the signal into $[0,2\pi]$, since the function $g$ returns always a positive value.\\
There are also interesting variations to the separable sigmoid, properly designed to work using a complex-valued network on real-valued data. But, for this reason, they are functions with values in $\mathds{R}$ and not in $\mathds{C}$, and so we won't go through them in this work.\\
After the advent of the ReLU activation functions, two designs where developed in this fashion, the \texttt{$\mathds{C}$ReLU} and the \texttt{$z$ReLU}:
These functions also share the nice property of being holomorphic in some regions of the complex plane: \texttt{$\mathds{C}$ReLU} in the first and third quadrants, while \texttt{$z$ReLU} everywhere but the set of points $\left\{\Re(z)>0,\,\Im(z)=0\right\}\cup\left\{\Re(z)=0,\,\Im(z)>0\right\}$.
\subsection*{Phase-preserving Activations}
...
...
@@ -209,7 +209,8 @@ It is called in this way because it is equivalent to applying the sigmoid to the
Unlike the sigmoid and its separable version, the siglog projects the magnitude of the input from the interval $[0, \infty)$ to $[0,1)$. The authors of this proposal suggested also the addition of a couple of parameters to adjust the \textit{scale}, $r$, and the \textit{steepness}, $c$, of the function:
The main problem with \textit{siglog} is that the function has a nonzero gradient in the neighborhood of the origin of the complex plane, which can lead to gradient descent optimization algorithms
to continuously stepping past the origin rather then approaching the point and staying here.\\
The main problem with \textit{siglog} is that the function has a nonzero gradient in the neighborhood of the origin of the complex plane, which can lead to gradient descent optimization algorithms
to continuously stepping past the origin rather then approaching the point and staying here.\\
For this reason, an alternative version have been proposed, this time with a better gradient behavior (approaching zero as the input approaches zero), that goes under the name of \texttt{iGaussian}.
This activation is basically an inverted gaussian (and this is the reason for which it is more smooth around the origin) and so depends only on one parameter, i.e. its standard deviation $\sigma$.\\
...
...
@@ -233,12 +234,20 @@ We will see, in our applications, that the cardioid effectively allows complex n
\newlabel{fig:cmplx_convolution}{{3.4}{xxiii}{Implementation details of the Complex Convolution (by \cite{trabelsi2018deep}).\relax}{figure.caption.19}{}}
\@writefile{toc}{\contentsline{section}{\numberline{A.2}Stationary points of a real-valued function of a complex variable \cite{Messerschmitt_stationary_points}}{xxviii}{section.A.2}\protected@file@percent }
\@writefile{toc}{\contentsline{section}{\numberline{A.2}Stationary points of a real-valued function of a complex variable \cite{Messerschmitt_stationary_points}}{xxx}{section.A.2}\protected@file@percent }
\contentsline{section}{\numberline{A.2}Stationary points of a real-valued function of a complex variable \cite{Messerschmitt_stationary_points}}{xxviii}{section.A.2}%
\contentsline{section}{\numberline{A.2}Stationary points of a real-valued function of a complex variable \cite{Messerschmitt_stationary_points}}{xxx}{section.A.2}%
