Let $N$ be an $n$-dimensional compact oriented manifold without boundary, and let $X$, $Y$ be compact oriented submanifolds of complementary dimensions $k$ and $n - k$, with boundary but such that $X \cap \partial Y = \varnothing$ and vice-versa, and which meet transversally. This answer to another question states that, under these conditions, it should be possible to define an intersection number of $X$ and $Y$ (as is intuitively clear), dual to the cup product of appropriate cohomology classes, but I am having some trouble following the reasoning and actually proving this.
To be precise, I think the cohomology classes should lie in $H^{n-k}(N,\partial X)$ and $H^k(N,\partial Y)$, so their product yields an element of $H^n(N, \partial X \cup \partial Y) \cong H_0\left(N - (\partial X \cup \partial Y)\right) \cong \mathbb{Z}$. Now, $X$ obviously defines an element of $H_k(N,\partial X)$, but this is dual to $H^{n-k}(N - \partial X)$, not $H^{n-k}(N,\partial X)$, which is instead dual to $H_k(N - \partial X)$, and I don't see how $X$ would define a class there (and same for $Y$).
Vice-versa, I could think of both $X$ and $Y$ as defining elements of $H_*(N,\partial X \cup \partial Y)$, dual to $H^{n-*}\left(N-(\partial X \cup \partial Y)\right)$, but then the cup product would lie in $H^n\left(N - (\partial X \cup \partial Y)\right) \cong H_0(N, \partial X \cup \partial Y) = 0$, so I just get zero.
What am I getting wrong?