I want to calculate $\hat{X}$ from a simple matrix equation, \begin{equation} \hat{A}\hat{X} = \hat{B} \end{equation} in which $\hat{A}$, $\hat{B}$, and $\hat{X}$ are all square $N\times N$ matrices. However, the matrices are very large and solving for $\hat{X}$ is practically impossible.
Considering that in the end, I want to use $\hat{X}$ to get $\vec{u} = \vec{P}\hat{X}$, is there anyway to directly solve for $\vec{u}$? ($\vec{P}$ and $\vec{u}$ are both $1\times N$ vectors.)
You want to find $$ P A^{-1} B $$ You can do this by first finding $$ Q = P A^{-1} $$ and then finding $QB$.
Finding $Q$ amounts to solving $$ QA = P $$ or, equivalently $$ A^t Q^t = P^t. $$
Solving for $Q^t$ in this situation using a linear-equation solver is generally much faster than inverting $A$.
Approximate matlab code (untested):