What conditions must $ a $ and $ b $ satisfy in order for the curve $$ \gamma(t) = \left( A \cos(\sqrt{a} t),B \cos \! \left( \sqrt{b} t \right) \right) $$ to be dense in the rectangle $ [- A,A] \times [- B,B] $?
Hint: Consider the system of harmonic oscillators \begin{align} x'' + a x & = 0. \\ y'' + b y & = 0. \end{align} Examine the possibility of integral curves in $ {\mathbb{R}^{4}}(x,x',y,y') $ being dense in the torus.
It is easier to "see these kind" of problems in polar coordinates. Your problem is equivalent to:
Oscillator 1: $$ \begin{eqnarray} r_1' &=0,\\ \theta_1' &=-\omega_1. \end{eqnarray} $$
Oscillator 2: $$ \begin{eqnarray} r_2' &=0\\ \theta_2' &=-\omega_2. \end{eqnarray} $$
Now we care only on the "standard equations" of the torus $$ \theta_1' =-\omega_1, \qquad \theta_2' =-\omega_2. $$
We have:
To convince yourself of this fact I give some pointers (leaving the details to you):
If $\frac{\omega_2}{\omega_1}$ is rational, then there are integers $p,q$ such that $\omega_1=p\tau$, $\omega_2=q\tau$. The solutions are $\theta_1(t)=\theta_1(0)-\omega_1t$, $\theta_2(t)=\theta_2(0)-\omega_2t$. Note that a solution is periodic if and only if $\exists 0\leq T<\infty$ such that $\theta_i(T)=\theta_i(0)+-\omega_iT=\theta_i(0)+2\pi N_i$, where $N_i$ is an integer. If you substitute appropriately you find that $T=2\pi\frac{p}{\omega_1}=2\pi\frac{q}{\omega_2}$, which holds if one only if our hypothesis is true.
The case where $\frac{\omega_2}{\omega_1}$ is irrational is just a bit tricker. The naive reasoning is that if the solutions can't be periodic then they must fill the torus in infinite time, which is indeed true. The proof that I know makes use of the Poincarè map and of maps of the circle into the circle ($S^1\to S^1$) but I don't know if you are familiar with those things. I believe that it is better to give you a nice reference: V.I. Arnold, Ordinary Differential Equations, $\S 24$, section $3$, page $217$.