The equation $$ x''(t)+\cos t \,x(t)=0 \quad (1) $$ can be transformed to the system: $$\vec{x}'= \begin{pmatrix} 0 & 1\\ -\cos t & 0 \end{pmatrix} \vec{x}=A(t) \cdot x(t) $$ with minimum period $T=2\pi$. Let $\mu_1,\mu_2$ be its characteristic values. A theorem gives: $$\mu_1\mu_2=\exp\Bigg\{\int_0^{2\pi} tr(A(t))dt\Bigg\}=1 \quad (2) $$ Therefore, the Wronskian of any two linearly independent solutions satisfies: $$ W(t+2\pi)=W(t) \quad (3) $$
Does $(3)$ imply that all solutions are bounded and thus we have asymptotic stability? If not, in what way could we use $(2)$ to determine $(1)$'s stability?
Hint :
If you want to make a conclusion about the stability of the given system and thus the initial equation via Floquet Theory (as your initial approach), then there is a theorem, that states that if for a multiplier $\mu_j$ for your given system, it is $|\mu_j| <1$, then the system is unstable. But, that's true, if you can prove that :
$$\mu_1\mu_2 = 1 \Rightarrow |\mu_1\mu_2| = 1 \Leftrightarrow |\mu_1| = \frac{1}{|\mu_2|} <1, \; \text{if} \; \mu_1, \mu_2 \neq 1$$
In order to conclude that, use the case of the characteristic matrix being periodic, thus $\Phi(t+T) = \Phi(t) \Rightarrow \Phi(2 \pi) = \Phi(0)E \Rightarrow E= \Phi(0)^{-1}\Phi(2\pi)$.
Now, the characteristic values will be the eigenvalues of the matrix $E$. You can calculate them (or approximate them) and conclude if $\mu_1,\mu_2 \neq 1$.
You can find more information and elaborations (proofs etc) about that theorem (which also states 2 cases about stability and asymptotic stability) and Floquet Theory in general, here.
Graphs :
(A phase portrait for a certain $t$) For a simple case of time $t$ such that $\cos t = -1$, the system has the image of the phase portrait :
$\qquad \qquad \qquad \quad$
which is a saddle, thus unstable.
Now, a sample solution family by sampling some initial values for the solution of the given equation and its derivative, one can see the unstability :
$\qquad \qquad \qquad \qquad$