The following difference equation is given, where $u$ is the unit step function. I want to calculate the system function $H(z)$. I started the calculation by performing the z-transform.
\begin{align*} y[n]&=ax[n]+5u[n] \\ Y(z)&=aX(z)+\frac{5}{1-z^{-1}} \\ H(z)=\frac{Y(z)}{X(z)}&=a+\frac{5}{1-z^{-1}}\frac{1}{X(z)} \end{align*}
How can I deal with the $\frac{1}{X(z)}$ on the right hand side? Is it even possible to calculate $H(z)$?
Actually, you can't, once it is a nonlinear difference equation:
A system is linear iff it is both additive and homogeneous. So, if at least one of this properties don't hold for the system, then it is nonlinear.
Additivity test:
Consider three inputs $x_1[n]$, $x_2[n]$ and $x_3[n]=x_1[n]+x_2[n]$. The system will be additive if $y(x_1+x_2)[n]=y(x_1)[n]+y(x_2)[n]$.
$$ \begin{align} y(x_1)[n]&=ax_1[n]+5u[n]\\ y(x_2)[n]&=ax_2[n]+5u[n]\\ \end{align} $$
$$y(x_1)[n]+y(x_2)[n]=\color{blue}{ax_1[n]+ax_2[n]}\color{red}{+10u[n]}$$
$$y(x_1+x_2)[n]=a(x_1[n]+x_2[n])+5u[n]=\color{blue}{ax_1[n]+ax_2[n]}\color{red}{+5u[n]}$$
Clearly, as $y(x_1+x_2)[n]\neq y(x_1)[n]+y(x_2)[n]$, the system is not additive. This is sufficient to classify it as nonlinear, but for the sake of explanation, let's keep on.
Homogeneity test:
Consider the inputs $x_1$ and $x_2=kx_1$, where $k$ is a constant. The system is homogeneous if $y(kx_1)[n]=ky(x_1)[n]$.
$$ \begin{align} y(x_1)[n]&=ax_1[n]+5u[n]\\ y(x_2)[n]&=y(kx_1)=\color{blue}{akx_1[n]}\color{red}{+5u[n]}\\ \end{align} $$
$$ky(x_1)[n]=\color{blue}{akx_1[n]}\color{red}{+5ku[n]}$$
So, as $y(kx_1)[n]\neq ky(x_1)[n]$, the system is not homogeneous.