With a single $X$ and $J$, you can't. For instance
$$
\begin{bmatrix} 1\\0\end{bmatrix} ^T\begin{bmatrix} a&b\\ c&d\end{bmatrix} \begin{bmatrix} 1\\0\end{bmatrix}=a.
$$
You are completely free to choose $b,c,d$.
Conceptually, $J$ is a single number and $P$ is determined by $n^2$ numbers.
With a single $X$ and $J$, you can't. For instance $$ \begin{bmatrix} 1\\0\end{bmatrix} ^T\begin{bmatrix} a&b\\ c&d\end{bmatrix} \begin{bmatrix} 1\\0\end{bmatrix}=a. $$ You are completely free to choose $b,c,d$.
Conceptually, $J$ is a single number and $P$ is determined by $n^2$ numbers.