I read this book from Eduardo Sontag: It provides multiple concepts on the control area, which I find very impressive. While reading about the dual concepts of controllability/observability of nonlinear systems, I see a potential bridge between a previous concept I learned in linear systems, the zero dynamic. For linear systems, the zero dynamics correspond to a subset $N \subset X$ in which the output vector $C \, x(t)$, for matrix $C \in \mathbb{R}^{p \times n}$ is 0. The transmission zeros correspond to the root complex numbers $s$ of the following characteristic polynomial of the following generalized eigen-problem.
\begin{equation} \mbox{det}\begin{pmatrix} s I - A & -B \\ C & D \end{pmatrix} = 0 \end{equation}
$\mathbf{Question}$: How can I relate the above problem with Hautus condition of pencil ranks $\mbox{rank}(\begin{bmatrix} s I - A & B \end{bmatrix})$ or $\mbox{rank}\left(\begin{bmatrix} s I - A \\ C \end{bmatrix}\right)$ ?
The two are not directly related. Also note that one is an eigenvalue problem, which returns a list eigenvalues (and eigenvectors) and the other is a rank check which is boolean true or false (though normally numerically this is calculated using the singular value decomposition and uses a threshold on the smallest singular value).
There are multiple ways to derive the generalized eigenvalue problem. One way would be to start in the time domain
\begin{align} \dot{x}(t) &= A\,x(t) + B\,u(t), \\ y(t) &= C\,x(t) + D\,u(t), \end{align}
and finding $x(t)$ and $u(t)$ such that $y(t)=0$. This can be done by taking the Laplace transform and factoring out $X(s)=\mathcal{L}\{x(t)\}(s)$ and $U(s)=\mathcal{L}\{u(t)\}(s)$, which yields
$$ \begin{bmatrix} 0 \\ 0 \end{bmatrix} = \begin{bmatrix} A - s\,I & B \\ C & D \end{bmatrix} \begin{bmatrix} X(s) \\ U(s) \end{bmatrix}, $$
which is equivalent to the generalized eigenvalue problem of the transmission zeros.
Another derivation would be with the help of the Schur complement, though which the matrix inside the determinant can also be written as
$$ \begin{bmatrix} A - \lambda\,I & B \\ C & D \end{bmatrix} = \underbrace{ \begin{bmatrix} I & 0 \\ C \left(A - \lambda\,I\right)^{-1} & I \end{bmatrix} }_{U} \underbrace{ \begin{bmatrix} A - \lambda\,I & 0 \\ 0 & C\left(\lambda\,I - A\right)^{-1}B+D \end{bmatrix} }_{M} \underbrace{ \begin{bmatrix} I & \left(A - \lambda\,I\right)^{-1}B \\ 0 & I \end{bmatrix} }_{V}. $$
For $\lambda$ not an eigenvalue of $A$ the matrices $U$ and $V$ are well defined, square and always full rank (due to the block triangular structure and the identity matrices along the diagonal). So the only way for $(1)$ to be true requires that $\det(M) = 0$ and because of the block diagonal structure of $M$ this is satisfied for $\det(C\left(\lambda\,I - A\right)^{-1}B+D) = 0$. For more details see my answer here.
The only relation between generalized eigenvalue problem of the transmission zeros and the Hautus tests for observability and controllability is that if the system is not observable or controllable then one (or more) of the eigenvalues of $A$ are also transmission zeros. This is equivalent to pole zero cancellation when looking at transfer functions instead of the state space model.