Assume a function $f:\mathbb{W}\rightarrow\mathbb{W}$ satisfying, $$(f(2n+1))^2-(f(2n))^2=6f(n)+1$$ and it is given than $f(2n)\geq f(n), \forall n\in \mathbb{W}$ where $\mathbb{W}$ is the set of whole numbers.
I started by putting $n=0$ which gave, $$(f(1))^2-(f(0))^2=6f(0)+1$$ which can be rewritten as, $$(f(1))^2-(f(0)+3)^2=-8$$
Only whole numbers which satisfy this are, $f(1)=1,f(0)=0$
The function may be a polynomial type, but setting $f(x)=ax²+bx$ doesn't really helped since degree of polynomials formed both side doesn't match and similarly cubic did so.
I don't know how do I proceed next. Can someone help me solving this?
I'm not sure if we can quite get a full formula for $f$, but we can simplify the functional equation greatly and get an algorithmic procedure for determining the values $f$ outputs.
Let us first prove your assertion that $f(0) = 0$ and $f(1) =1$. If $n = 0$, then $[f(1)]^2 = [f(0)]^2 + 6f(0) + 1$. We can limit the choices on $f(0)$ and on $f(1)$ if we can find all of the $k \in \mathbb{W}$ such that $k^2 + 6k + 1$ is a square. $k = 0$ works, but if $k \geq 1$, $$ (k+1)^2 = k^2 + 2k + 1 < k^2 + 6k + 1 < k^2 + 6k + 9 = (k+3)^2, $$ so to have that $k^2+6k+1$ is a square, we must have $$ k^2 + 6k + 1 = (k+2)^2 = k^2 + 4k + 4 \implies 2k = 3, $$ which is impossible. So $k = 0$ is the only possible choice, immediately yielding that $f(0) = 0$ and $f(1) = 1$.
Now, notice that for all $n$, since $f$ is nonnegative, $$ 0 < 1 \leq 6f(n) + 1 = [f(2n+1)]^2 - [f(2n)]^2 = \left[f(2n+1) - f(2n)\right]\cdot\left[f(2n+1) + f(2n)\right] $$ and so we must have $f(2n+1) - f(2n) > 0$. But from $f(2n) \geq f(n)$ we must also have $$ [f(2n+1)]^2 = [f(2n)]^2 + 6f(n) + 1 < [f(2n)]^2 + 6f(2n) + 9 = \left[f(2n) + 3\right]^2, $$ implying $f(2n+1) < f(2n) + 3$. That is, we conclude $0 < f(2n+1) - f(2n) < 3$.
As we are in the integers, this is only possible if $f(2n+1)-f(2n) \in \{1,2\}$. But we cannot have $f(2n+1) - f(2n) = 2$, because then $2|(6f(n)+1)$ by the functional equation, and this is impossible. Therefore, $f(2n+1)-f(2n) = 1$ for all $n$. This reduces the functional equation to $$ 6f(n) + 1 = [f(2n+1)]^2 - [f(2n)]^2 = [f(2n)+1]^2 - [f(2n)]^2 = 2f(2n)+1, $$ or in other words, $f(2n) = 3f(n)$ and $f(2n+1) = 3f(n)+1$ for all $n$.
We can therefore determine the outputs of $f$ by pulling off as many powers of $2$ as we can from the input. For if we write $n = 2^r\cdot k$ for some $r,k \in \mathbb{W}$ with $k$ odd, then $$ f(n) = f(2^r\cdot k) = 3f(2^{r-1}\cdot k) = \cdots = 3^rf(k) = 3^r\left[f(k-1) + 1\right], $$ and we repeat the process by writing $k-1 = 2^s\cdot l$, etc. Notice that as we repeat, $f(n)$ becomes a sum of unique powers of 3. That is, I believe you can prove that $f(n)$ is the $n$th number in the sequence of nonnegative integers which have no $2$ within their ternary expansion (see A005836 in the OEIS). This assertion certainly checks out up to $n = 12$, which I checked by hand (and I am too lazy to check more).