I know that if $B \supset A$ are two rings and $B$ is integral over $A$, then $\sigma(B)$ is integral over $\sigma(A)$ where $\sigma : B \to C$ is any hom into a third ring.
I want to use that fact to prove that if $\hat{p}$ is a prime ideal in $A$ and $\hat{B}$ lies above $\hat{p}$, or in other words $\hat{B} \cap A = \hat{p}$, then $B/\hat{B}$ is integral over $A/\hat{p}$.
I know that the following diagram commutes:
$$ \require{AMScd} \begin{CD} A @>{\iota}>> B\\ @V\pi_AVV @V\pi_BVV \\ A/\hat{p} @>{\iota^*}>> B/\hat{B} \end{CD} $$ where $\pi$'s are natural surjections, and $\iota$ is inclusion, and $\iota^*$ is induced by $\iota$.
So we take $\sigma = \pi_B$. Then $\sigma(A) = \pi_B \circ \iota(A) = \iota^* \circ \pi_A(A)$.
But isn't $\iota^* \circ \pi_A(A) = B/\hat{B}$?
Your task is to prove that $B/\hat{B}$ is integral over $\iota^*(A/\hat{p})$. Since $\pi_A$ is surjective, $\iota^*(A/\hat{p})=\iota^* \circ \pi_A(A)=\sigma(A)$. So since you know $\sigma(B)=B/\hat{B}$ is integral over $\sigma(A)$, you're done. It is entirely irrelevant whether $\iota^* \circ \pi_A(A) = B/\hat{B}$.
In any case, the equation $\iota^* \circ \pi_A(A) = B/\hat{B}$ usually is not true, and I'm not sure why you think it is. For example, if $A$ and $B$ are domains and $\hat{p}=\hat{B}=0$, then $B/\hat{B}$ is just $B$ and $\iota^* \circ \pi_A(A)$ is just $A$, so they are only equal if $A$ was all of $B$ to begin with.