I want to calculate the annual instalment for a mortgage of $\$\:150 \,000$ to be repayed in 25 years at $3\%$ interest rate, but the formula I obtained is wrong. Can you help me please?
The equation describing a mortgage is the following: $$P_{t+1}=(1+r)P_t-I$$ where $P_t$ is the principal at time $t$, $r$ is the interest rate and $I$ the annual installment.
By applying the Zeta transform I obtain
$$zP(z)-zP_0=(1+r)P(z)-I$$ $$[z-(1+r)]P(z)=zP_0-I$$ $$P(z)=\frac{zP_0-I}{z-(1+r)}$$ $$\frac{P(z)}{z}=\frac{zP_0-I}{z[z-(1+r)]}=\frac{A}{z}+\frac{B}{z-(1+r)}=\frac{Az-A(1+r)+Bz}{z[z-(1+r)]}=\ldots$$
$$A+B=P_0$$ $$A=\frac{I}{1+r} \Rightarrow B=P_0-\frac{I}{1+r}$$
$$\ldots=\frac{I}{1+r} \frac{1}{z}+\left(P_0-\frac{I}{1+r} \right)\frac{1}{z[z-(1+r)]}$$
$$P(z)=\frac{I}{1+r} +\left(P_0-\frac{I}{1+r} \right)\frac{z}{z[z-(1+r)]}$$
$$P_t=\frac{I}{1+r} \delta(t)+\left(P_0-\frac{I}{1+r} \right)(1+r)^tH(t)$$
At time $T$ the principal must be $0$
$$P_T=\left(P_0-\frac{I}{1+r} \right)(1+r)^T=0$$
This formula must be wrong...
(Note: I do not know what a Zeta transform is.)
Indeed, if you start off with $P_0$ dollars to pay back after $0$ years, and $P_t$ is the amount to be paid back after $t$ years, and you pay back a constant amount $I$ each year, and $r$ is the interest rate compounded annually, then
$$P_{t+1}=(1+r) P_t - I.$$
This is a non-homogenous linear recurrence relation with constant coefficients and you can prove, for example with induction, that
$$\bbox[15px,border:1px groove navy]{P_t = (1+r)^t P_0 - I \sum_{k=0}^{t-1} (1+r)^k=(1+r)^t P_0-\frac{(1+r)^t-1}r I.}$$
Intuitive formulation. The initial loan compounds year for year, but each annual payback $I$ also compounds year for year and thus has to be subtracted from the payback $P_t$ that is due after $t$ years.
Now to your case: We have $r=0.03, P_0=150000$ and we want $P_{25}=0$. So $$P_{25}=0\iff 1.03^{25}\cdot 150000-\frac{1.03^{25}-1}{0.03}I=0$$
which is a linear equation with solution $I\approx 8614.18$ (in this case in dollars).
Of course computations are a bit more complicated if you pay back for example monthly or if interest compounds semi-annually etc. But there are many Websites (for instance https://www.calculator.net/loan-calculator.html) that do the computations for you.