Decomposition of abelian varieties up to isogeny

Question

Let $A_1,A_2,B_1,B_2$ be simple abelian varieties over a number field $k$. Suppose that $A_1\times A_2$ is $k$-isogenous to $B_1\times B_2$. Can we deduce that (up to reordering the factors) $A_1$ is $k$-isogenous to $B_1$ and $A_2$ is $k$-isogenous to $B_2$?