For $a \in\{1,2,\ldots,n-1\}$, I need to maximize the following: $$\!\left(a\sqrt{\frac{a}{a+1}}+\sqrt{\frac{a(n-a)}{(a+1)(n-a+1)}}\right) \left((n-a)\sqrt{\frac{n-a}{n-a+1}}+\sqrt{\frac{a(n-a)}{(a+1)(n-a+1)}}\right) $$ It seems that max value will occur at $a=1$. Since above is a symmetric function in variable $a,n-a$, max value will also occur at $a=n-1.$
I am not able to prove that maximum will occur at $a=1$. How to prove it?
Your assumption that the maximum will occur when $a=1$ is false so you can't prove it. Look at the this wolfram alpha plot (eg. I've set the n=100 so the plot is not 3D).