I have $$f(x) = x^3-2x^2-5x+6,\quad g(x)=x^2-2x-3$$
Then $f(x) = x(x^2-2x-3) +(-2x+6)$
So $hcf(f(x),g(x))=hcf((x^2-2x-3),(-2x+6))$
$x^2-2x-3=(-\frac{1}{2}x-\frac{1}{2})(-2x+6)+0$
So $hcf(f(x),g(x))=hcf((-2x+6),(0)) = 0$
Is this correct? If yes, then is the $0$ representing polynomials of degree $0$ or polynomials $h(x)$ such that $h(x)=0$?
When you get to zero, the highest common factor is the one you had just before. Here $2x-6=2(x-3)$ and $x-3$ is the highest common factor of the first polynomials.