Suppose we have a pair of adjoint founctors $S:\mathcal{H} \to \mathcal{G}$ and $T:\mathcal{G} \to \mathcal{H}$ where $\mathcal{G}$ and $\mathcal{H}$ are both additive categories, and $S$ is left adjoint to $T$ and $T$ is right adjoint to $S$ with an adjunction
$$\alpha(N,M):hom_{\mathcal{G}}(SN,M) \to hom_\mathcal{H}(N,TM)$$
Further, let $\mathcal{X} \subseteq \mathcal{G}$ and $\mathcal{Y} \subseteq \mathcal{H}$ be subcategories (full additive subcategories).
Let $f$ be a morphism from $N \to TM$ which factors through $Y$ for some $Y \in Obj\mathcal{Y}$. We have $f$ of he form $f=pq$
$$N \xrightarrow{p} Y \xrightarrow{q} TM$$
Now, it is claimed that
$$\alpha^{-1}(f)=\alpha^{-1}(pq)=\alpha^{-1}(p) \circ Sp$$
but I do not see why? An adjunction is just an isomorphism in the category of sets so it doesn't necessarily respect composition in any nice way. Is this a general fact about adjunctions I don't know or is there something more specific going on?
An adjunction is not just an isomorphism in the category of sets. It's actually a family of bijections $$\alpha(N,M):hom_{\mathcal{G}}(SN,M) \to hom_\mathcal{H}(N,TM),$$ for each $N\in Obj(\mathcal{H})$ and $M\in Obj(\mathcal{G})$, which must be natural with respect to $N$ and $M$, in the sense that this family defines a natural isomorphism $$\alpha:hom_{\mathcal{G}}(S\_,\_) \to hom_\mathcal{H}(\_,T\_)$$ in the category of functors $\mathcal{H}^{op}\times \mathcal{G}\to\mathbf{Set}$ (in your case you could replace $\mathbf{Set}$ with $\mathbf{Ab}$). This naturality condition is precisely the equality in your question (and its dual); see this question for more details.