There is a version of Poincaré duality (page 53 of Griffths-Harris) which reads:
Let $X$ be a closed oriented (connected) $n$-manifold. The intersection pairing $$H_k(X)\otimes H_{n-k}(X)\to \mathbb{Z}$$ yields a surjection $H_k(X)\to \mathrm{Hom}(H_{n-k}(X),\mathbb{Z})$, moreover any $x\in H_k(X)$ with $$x\otimes H_{n-k}(X)\mapsto 0$$ is torsion.
This points to the following decomposition: $$H_k(X)\cong\mathrm{Hom}(H_{n-k}(X),\mathbb{Z})\oplus \mathrm{Torsion}(H_k(X)).$$ The universal coefficient theorem gives us $$H^{n-k}(X)\cong \mathrm{Hom}(H_{n-k}(X),\mathbb{Z})\oplus\mathrm{Hom}(H_{n-k-1}(X),\mathbb{Q}/\mathbb{Z}).$$
Note that an isomorphism between the second summands above would recover the Poincaré duality isomorphism $\phi\colon H_k(X)\cong H^{n-k}(X)$ (but naturality seems an issue to me).
Edit 2: Thanks to the comments, I added a specific question and edited the text accordingly.
Question: How can one prove $\mathrm{Torsion}(H_k(X))\cong\mathrm{Hom}(H_{n-k-1}(X),\mathbb{Q}/\mathbb{Z})$ by assuming the theorem in blockquote (not assuming the map $\phi$)?