Let $p(x)=x^2+5x-3$ , $q(x)=4x^2+18x+4,$ and $s(x)=x^2+8x+2$.
I want to find two constants $a$ and $b$ such that $$s(x)=a \cdot p(x) +b \cdot q(x)$$
My attempt: $$ap(x)+bq(x) = ax^2+5ax-3a+4bx^2+18bx+4b = (a+4b)x^2+(5a+18b)x+4b-3a$$
I put this in the augmented matrix (the last column to the right is the coefficients of $s(x)$:
$\begin{bmatrix}1&4&1\\5&18&8\\-3&4&2\end{bmatrix}$.
I reduced this to echelon form $\begin{bmatrix}1&4&1\\0&2&-3\\0&-8&-1\end{bmatrix}$ and found out that the system is inconsistent, and has no solutions, meaning that there are no such $a$ and $b$ that satisify the given equation.
Is my solution correct?
Yes your way is fine, indeed with respect to the standard basis $\{1,x,x^2\}$ we have
and we need to solve
$$av_p+bv_q=v_s \iff \begin{bmatrix}-3&4\\5&18\\1&4\end{bmatrix} \begin{bmatrix}a\\b\end{bmatrix} =\begin{bmatrix}2\\8\\1\end{bmatrix}$$
and by the augmented matrix we obtain
$$\begin{bmatrix}-3&4&2\\5&18&8\\1&4&1\end{bmatrix}\to \begin{bmatrix}-3&4&2\\0&74&34\\0&16&5\end{bmatrix}$$
which has not solution.
As an alternative note that
$$\det\begin{bmatrix}-3&4&2\\5&18&8\\1&4&1\end{bmatrix}=-54+32+40-36+96-20=58 \neq 0$$
therefore the three vectors are linearly independent.