Here $A^{\dagger}$ is the Moore-Penrose pseudo-inverse and $\geq$ denotes the Loewner partial order for positive semidefinite matrices.
I know that the statement is true for non-singular matrices, but cannot find whether this extends to singular matrices, and continuity arguments seem to not apply here.
Any help will be greatly appreciated.
It isn't true. The example could be the following: $$ A=\begin{pmatrix}1&0\\0&1\end{pmatrix},\quad B=\begin{pmatrix}1&0\\0&0\end{pmatrix}. $$ Their pseudoinverse matrices are $$ A^+=A,\quad B^+=B. $$ Which implies $$ A\ge B,\quad B^+\not\ge A^+. $$