I need to apply the reduction of order into differential equation $$\cos^2xy''-6y=0$$where the first solution is of the form $$y_1=(\beta \tan^2x+1)$$ with $\beta \in R$.
I started solving equation with reduction of order, however I don't know how should I use the knowledge of "$\beta \in R$."
$$y_1=(\beta \tan^2x+1)$$ $$y_2=u\cdot y_1=u\cdot(\beta \tan^2x+1)$$ $$y_2'=(u\cdot y_1)'= \dot u(\beta \tan^2x+1)+u(2\beta \cdot \tan x\cdot \sec^2x)$$
$$y_2''= (\dot u(\beta \tan^2x+1)+u(2\beta \cdot \tan x\cdot \sec^2x))'= \ddot u (\beta \tan^2x+1)+ \dot u(4\beta \tan x \cdot \sec^2x) +u(2\beta \sec^4x+4\beta \tan^2x \sec^2x) $$
After inserting $y_2, y_2',y_2''$ into $\cos^2xy''-6y=0$ I got the form where any of the element of the equation want to reduce. Therefore, I would like to ask should I assume do with $\beta$, can I assume that $\beta$ equals for instance 1?
What you are doing is too complicated. You know $y_1$ is a solution, you just use that knowledge:
$$y_2=u y_1$$
$$y_2'=u'y_1+u y_1'$$
$$y_2''=u''y_1+2u' y_1'+u y_1''$$
Now substitute in the equation:
$$\cos^2 x~(u''y_1+2u' y_1'+u y_1'')-6 u y_1=0$$
Using the fact that:
$$\cos^2 x~y_1''-6y_1=0$$
We have:
$$\cos^2 x~(u''y_1+2u' y_1')=0$$
Now you only need to compute $y_1'$ (already done) and take $u'=v$ as the new function.
Edit
However, $y_1$ is not a solution for any $\beta$, the OP was right.
To find $\beta$ we need to substitute just $y_1$ in the equation and find for which value of $\beta$ it is correct.
In other words, you need to simplify:
$$\cos^2 x ( \beta \tan^2 x+1)''-6 ( \beta \tan^2 x+1)=0$$
I haven't done this by hand, but Wolfram Alpha gives $\beta=3$.