Suppose that $M$and $N$ are manifolds with nonempty boundary and that $f,g : M \to N$ with $f(\partial M), g(\partial M) \subseteq \partial N$. Also, suppose that $f \simeq g$ and that as maps $f|_{\partial M}, g|_{\partial M} : \partial M \to \partial N$, we have $f|_{\partial M} \simeq g|_{\partial M}$.
Is it then true that $f$ can be homotoped to $g$ with a homotopy $H : M \times I \to M$ with $H_t(\partial M) \subseteq \partial N$ for all $t \in I$? In particular, I am interested in the case where $M$ and $N$ are handlebodies.
No. Consider a manifold $M$ with connected boundary $\partial M$ so that $\pi_1(M,\partial M) $ has more than one element. Let f,g:I\rightarrow M represent distinct elements of $\pi_1(M,\partial M) $ . They satisfy the hypotheses of your question but cannot be homotoped to each other keeping the boundary of $I$ in $\partial M$.