My Question is How to proceed with the question. I mean there are two ways:- Either one could apply the scalar triple product and solve the determinant equated to 0 which has provided me with the relation of a Geometric Progression. Or We Can Multiply 2 of the vectors by some variables and add them to form the third vector and solve accordingly but consider multiplying the first and the third vector given and no solution can be obtained. Can Someone Explain to me the error in my work in an elementary fashion? I'd be really grateful
(Please excuse me for being unable to attach a pic for further explanation as I am new on the site and not familiar with the homework rules)
What Went Wrong
Okay, the problem is that in fact the vectors $(a,a,c)$ and $(c,c,b)$ are in the same one-dimensional subspace (that is, they are scalar multiples of each other).
Your correct solution tells us that the numbers are in geometric progression, and in particular we have $c = \lambda a$, $b = \lambda c$ for some scalar $\lambda$. Therefore, we have $$ (a,a,c) = (a,a,\lambda a) = a(1,1,\lambda), $$
and $$ (c,c,b) = (c,c,\lambda c) = c(1,1,\lambda). $$ It follows that any linear combination of $(a,a,c)$ and $(c,c,b)$ will lie in the one-dimensional subspace $\langle (1,1,\lambda)\rangle$, which clearly doesn't contain $(1,0,1)$.
How to Fix It
You can salvage your second approach by using a different pair of vectors. In my opinion, this is much more elegant than using matrix determinants, which are unnecessarily complicated. Since the vectors are coplanar, there exist scalars $\lambda, \mu$ such that $$ \lambda(a,a,c) + \mu(1,0,1) =(\lambda a + \mu, \lambda a, \lambda c + \mu) = (c,c,b). $$ Since the first two entries of the middle vector are equal, we have $\mu = 0$, so $$ (\lambda a, \lambda a, \lambda c) = (c,c,b). $$ It follows immediately that $c = \lambda a$ and $b = \lambda c$.