Question : Find the value of x so that A[-1,3,4] , B[-2,3,-1] and C[-5,6, x] all lie on a plane that contains the origin Answer : -7
The confusion for me here is more so what the question is looking for / the method it wants me to take in order to arrive at the desired answer, I would simply turn to a solutions manual or a service like Photomath however the former does not show process and the latter only works with numerical problems
On
A plane in 3d space can be defined by any triple of pairwise distinct points $A,B,O$, which are not collinear. It can be parameterized in the sense that every point $P$ on the plane can be expressed uniquely as $$ p = o + \lambda (a-o) + \mu (b-o)$$ with $\lambda,\mu$ being real numbers. Note that I switched from point notation $A,B,O,P$ to vector notation $a,b,o,p$. The question asks you to find that $x$ for which the point $C$ lies on the plane, which means that you replace $p$ in the above equation with $c$. Moreover note that $o = 0$ in your case.
Summarizing, you have to solve the linear system of equations $$\begin{align*} -5&=-y-2z\\ 6&=3y+3z\\ x&=4y-z\end{align*}$$
On
The normal to the plane containing $\vec A$, $\vec B$, and the origin $\vec 0$ is $$\vec n=(\vec A-\vec0)\times(\vec B-\vec 0)=\langle-1,3,4\rangle\times\langle-2,3,-1\rangle=\langle-15,-9,3\rangle$$ So the equation of the plane containing the $3$ points is $$\vec n\cdot\vec r=\vec n\cdot \vec r_0=-15x-9y+3z=0$$ Where $\vec r=\langle x,y,z\rangle$ and we have taken a point in the plane as $\vec r_0=\vec0=\langle0,0,0\rangle$. Then $$\vec n\cdot\vec C=(-15)(-5)+(-9)(6)+(3)(x)=21+3x=0$$ So $x=-7$
On
the method it wants me to take in order to arrive at the desired answer
This might depend. Sometimes, homework assignment force a specific way or approach to solve a problem. Sometimes to get familiar with a specific topic, sometimes not to bust a teacher's head.
In your case, the statement is that $A$, $B$ and $C$ are not linear independent, i.e. their equation $$\alpha A + \beta B +\gamma C = 0$$ has solution(s) in real numbers $\alpha$, $\beta$ and $\gamma$ where not all of them are zero. One way to find this is to form a $3\times 3$ matrix of the vectors (does not matter whether you use them for columns or rows) and solve $$\det M=0$$ You can also understand this as linear system of 3 equations with 3 unknowns $\alpha$, $\beta$ and $\gamma$ that has to be solved dependent on a parameter $x$, i.e. $\alpha$, $\beta$ and $\gamma$ are actually functions of $x$, and you are searching $x$'s that permit non-trivial solution(s).
Hint
A plane that contains the origin has equation $aX+bY+cZ=0$. Substitute the coordinates of the given points and find the value of $x$ for which the system of the three equations in $a,b,c$ has a solution.