find the value of c such that the system will have periodic solution $$\overrightarrow x'(t)=\begin{pmatrix} 3 & -9\\ c & -3\\ \end{pmatrix}\overrightarrow x(t) $$
I found the eigenvalues det(A-λt), $$λ= \pm 3 \sqrt{1-c}$$ and then I tried to find the eigenvector for the positive eigenvalue $$\begin{pmatrix} 3-3 \sqrt{1-c} & -9\\ c& -3-3 \sqrt{1-c}\\ \end{pmatrix} $$ multiply with $$\begin{pmatrix} V_1\\ V_2\\ \end{pmatrix} $$ equal $$\begin{pmatrix} 0\\ 0\\ \end{pmatrix} $$ I am not sure how to find three unknowns here and not sure what the question means by periodic solutions.
Here's a different approach. Again we'll write $A$ for the system matrix. Begin by conjugating $A$ with an elementary matrix to obtain a matrix with zeros on the diagonal: $$ \left(\begin{array}{cr} 1 & 0 \\ -1/3 & 1\end{array}\right) \left(\begin{array}{cr} 3 & -9 \\ c & -3\end{array}\right) \left(\begin{array}{cr} 1 & 0 \\ +1/3 & 1\end{array}\right) = \left(\begin{array}{cr} 0 & -9 \\ c-1 & 0\end{array}\right) $$
Conjugate this with a diagonal matrix in order to make the off diagonal elements equal with opposite signs: $$ \left(\begin{array}{cr} \sqrt{c-1} & 0 \\ 0 & 3\end{array}\right) \left(\begin{array}{cr} 0 & -9 \\ c-1 & 0\end{array}\right) \left(\begin{array}{cc} 1/\sqrt{c-1} & 0 \\ 0 & 1/3\end{array}\right) = \left(\begin{array}{cc} 0 & -3\sqrt{c-1} \\ 3\sqrt{c-1} & 0\end{array}\right) $$
Writing $\omega$ for $3\sqrt{c-1}$, we see that your system matrix is conjugate to the skew matrix $$\Omega =\left(\begin{array}{cr} 0 & -\omega \\ \omega& 0\end{array}\right)$$
So the entries in the solution vector to your differential equation are linear combinations of the entries in the solution vector to the equation $$ \vec{y}'(t) = \Omega\; \vec{y}(t) $$ It's not hard to see that $$ \vec{y(t)} = \left(\begin{array}{c} \cos(\omega t) \\ \sin(\omega t) \end{array}\right) $$ is a solution, and it will be a periodic solution if $\bf\omega$ is real. Which holds whenever $c$ is greater than 1.