Let $f,g$ be arrows in an abelian category such that the composite $gf$ is defined and is given by the zero arrow. I shall try to find a definition for the quotient $\ker g /\operatorname{im} f$, which coincides with the usual notion for modules.
$gf=0$ means that $\operatorname{im}f\le\ker g$ as subobjects of $\operatorname{cod}f=\operatorname{dom}g$, so there is a (monic) arrow $u$ satisfying $\ker g\circ u=\operatorname{im}f$. I would define $\ker g/\operatorname{im}f$ as the (codomain of the) cokernel of $u$. Is that correct?
Then I have to show that $\ker g/\operatorname{im}f$ is isomorphic to the dual object $\operatorname{coim}g/\operatorname{coker}f$, i.e. the kernel of the canonical epimorphism $\operatorname{coker}f\to\operatorname{coim}g$. I have no idea how to show this. All I found out is that I have canonical arrows from $\operatorname{Ker}g$ to the object $\operatorname{coim}g/\operatorname{coker}f$ and from $\ker g/\operatorname{im}f$ to the cokernel of $f$.
As a next step (this is not an exercise) I would like to show, that also the Homology functors coincide depending on which of the two object one defines as the homology.
Name our arrows for now as $${\rm im\,}f\overset{i_1}\longrightarrow{\rm ker\,}g\overset{i_2}\longrightarrow B \overset{p_1}\longrightarrow{\rm coker\,}f\overset{p_2}\longrightarrow {\rm coim\,}g$$ and $\ {\rm ker\,}g\overset u\longrightarrow {\rm coker\,}i_1$, $\ {\rm ker\,}p_2\overset v\longrightarrow {\rm coker\,}f$.
We want to prove that $\ {\rm coker\,}i_1={\rm ker\,}p_2$.
(Let me write compositions from left to right.)
Note that both pairs $(i_1i_2,\,p_1)$ and $(i_2,\,p_1p_2)$ form short exact sequences.
As you already observed, by the kernel and cokernel properties, we obtain an arrow $d:{\rm coker\,}i_1\longrightarrow{\rm ker\,}p_2$.
Then the idea is to prove that $udv$ is the canonical deocmposition of $i_2p_1$, i.e. that $u={\rm coim}(i_2p_1)$ and $v={\rm im}(i_2p_1)$, thus it will follow that $d$ is an isomorphism.
Finally, e.g. $u={\rm coim}(i_2p_1)$ is equivalent, by duality, to $i_1={\rm ker}(i_2p_1)$, which can be easily verified.