$$y'(t)=-a_0(a_1 - b\beta(t))\cos(wt)$$ $$\beta'(t)= -a_0b y(t)\cos(wt)$$ i) is it non linear differential equation .I am skeptical about this one but the $\beta(t)\cos wt $ on the RHS proves it.
ii)How to solve this equation ?is there any special method?
First differentiate the second equation w.r.t the independent variable $t$. This gives an equation of the form $\beta''(t) = f(y, y')$. Eliminate the $y'$ from $f$ using first equation and eliminate the $y$ from $f$ using the second equation. This gives the following linear, second order ODE for $\beta(t)$: $$\beta''(t) + \omega \tan(\omega t) \beta'(t) + c^2 \cos^2(\omega t) \beta(t) = d \cos^2(\omega t).$$
Where the constants $c:=a_0 b$ and $d:= a_0^2 a_1 b$ are defined for brevity.
In order to solve this non-constant coefficient ODE it is useful to employ the change of independent variable $x:=c \sin(\omega t)$, after applying the chain rule the ODE becomes $$c^2 \omega^2 \cos^2(\omega t)\beta''(x) + c^2 \cos^2(\omega t) \beta(x) = d \cos^2(\omega t),\\ \Rightarrow \beta''(x) + \frac{\beta(x)}{\omega^2} = \frac{d}{c^2 \omega^2}.$$
The above equation is now a constant coefficient ODE. This can easily be solved to give $$\beta(x) = A \cos\left(\frac{x}{\omega}\right) + B \sin\left(\frac{x}{\omega}\right) + \frac{d}{c^2}.$$ In which $A$ and $B$ are arbitrary constants. Reverting back to $\beta(t)$ we get the solution of the ODE as $$\beta(t) = A \cos\left(\frac{c \sin(\omega t)}{\omega}\right) + B \sin\left(\frac{c \sin(\omega t)}{\omega}\right) + \frac{d}{c^2}.$$ The second solution, viz, $y(t)$ can now be found by rearrangement of the second ODE in the post.