How can I calculate z transfer function of the state space model when model includes vectors. For example what is z transfer function of following model when we assume that first state is the system output.
$x(k+1)=\begin{bmatrix} \ -a&A\\-b&0\end{bmatrix}x(k)+\begin{bmatrix} \ c\\d\end{bmatrix}u(k)$
You are starting of with a discrete state space model of the form
$$ \begin{array}{c} x(k+1) = A\,x(k) + B\,u(k) \\ y(k) = C\,x(k) + D\,u(k) \end{array} \tag{1} $$
The z-transfer function of that state space model can be obtained using
$$ G(z) = C\,(z\,I-A)^{-1}B + D. \tag{2} $$
You can derive this using that $x(k+1) = z\,x(k)$, so the first line from $(1)$ can be written as $(z\,I-A)\,x(k) = B\,u(k)$. Solving that for $x(k)$ gives $x(k) = (z\,I-A)^{-1}B\,u(k)$. Equation $(2)$ van now be obtained by substituting this for $x(k)$ into the expression for $y(k)$ in $(1)$.
What the $A$ and $B$ matrices should be in your example should be clear from your question. Since you stated that the output $y(k)$ is the first component of $x(k)$, then $C = \begin{bmatrix}1 & 0 & \cdots & 0\end{bmatrix}$ and $D=0$.