Given $t > 0$, let the function $f : \Bbb R_0^+ \times \Bbb R_0^+ \times[0,1]\times[-1,0]\to \Bbb R$ be defined as
\begin{align} &f(x,y,\rho_1,\rho_2)=\frac{(t+1)\left(x\left(1-\rho_1^2-\rho_2^2\right)+1\right)}{t+1-\left(\sqrt{y}+\sqrt{x}\rho_1\right)^2} \end{align}
I am interested in finding
$$\underset{x, y,\rho_1, \rho_2}{{\operatorname{argmax}}} f (x,y) \text{ such that the correlation coefficients } 0 \le \rho_1 \le 1, -1\le\rho_2\le 0, \text{ where } \rho_1^2 + \rho_2^2\le1 \text{ and } \rho_2 = -\frac{x+y + 2\sqrt{xy}\rho_1}{2\sqrt{tx}}$$
$0\le x\le x_1$, and $0\le y\le y_1$, where $x$ and $y$ are fixed.
Motivation
This is a power allocation problem for a telecommunication problem in which $t$ is the power of some noise which is fixed and non-negative, $x$ and $y$ are non-negative input powers subject to the power constraints $x_1$ and $y_1$, respectively. Here, it is assumed that $y$ is available for generating $x$. Therefore, the transmitter can correlate $x$ with $y$ to get some gains. $\rho_1$ is the correlation coefficients between some part of $y$ and $\rho_2$ is the correlation coefficients between another part of $y$ which has power $t$. Both $\rho_1$ and $\rho_2$ can be chosen by the user. We also know that the denominator is positive, this follows from the condition that $\rho_1^2+\rho_2^2\le1$.
You can reduce the equation to single variable $x$ by : $$u(x) = -\rho_2\times 2\sqrt{t x} -(1-\rho_1^2) x = (\sqrt{y}+\sqrt{x}\rho_1)^2$$.
Hence
\begin{align} &f(x,y)=\frac{(t+1)\left(x\left(1-\rho_1^2-\rho_2^2\right)+1\right)}{t+1-u(x)^2} \end{align}
\begin{align} &f(x,y)=\frac{(t+1)\left(x\left(1-\rho_1^2-\rho_2^2\right)+1\right)}{t+1+(\rho_2\times 2\sqrt{tx} +(1-\rho_1^2) x)} \end{align}
Hence $f(x,y) = f(x)$
Since $\rho_1,\rho_2,t$ are all fixed, the above is a function of one variable $x$. Now differntiate $f(x)$ and set to $0$ to obtain $x$ or check if the solution is at boundary of range of $x$.
If possible, tell me more about the physical significance or practical significance or operational significance of this problem you mentioned in the question more. Edit the question rather than commenting.
[EDIT]: Since as per comments below $\rho_2$ also varies, we have:
\begin{align} &f(x,y)/(t+1)=\frac{1+x(1-\rho_1^2) - x \rho_2^2}{1+x(1-\rho_1^2) + t+(\rho_2\times 2\sqrt{tx})} \end{align}
But we need $$-\rho_2\times 2\sqrt{t x} -(1-\rho_1^2) x = (\sqrt{y}+\sqrt{x}\rho_1)^2.$$ Since RHS is $\geq x \rho_1^2$ $\implies -\rho_2\times 2\sqrt{t x} \geq x \implies \rho_2 \leq -0.5\sqrt{x/t}$
Let $\rho_2 = -0.5\delta\sqrt{x/t}$, we get, for $\delta \geq 2\sqrt{\frac{t}{x}}$
\begin{align} &f(x,y)/(t+1)=\frac{1+x(1-\rho_1^2) - 0.25 \delta^2 x^2/t}{1+x(1-\rho_1^2) + t-\delta x} \end{align}
\begin{align} &f(x,y)t/(t+1)=\frac{t(1+x(1-\rho_1^2)) - 0.25 \delta^2 x^2}{1+x(1-\rho_1^2) + t-\delta x} \end{align}
Rest depends on relation between $x,t$, for example, If $\rho_2 = -0.5\delta\sqrt{x/t} \leq -1$ i.e.,$\sqrt{x/t} \geq 2$ i.e., $x \geq 4t \implies \rho_2 = -1$.
Solve the above equation range by range.
Use the below inequality: For $a>0,b>0,c>0,d>0$, $(a-b)/(c-d) \leq a/c \iff c(a-b) \leq a(c-d) \iff -bc \leq -ad \iff b/d \geq a/c$
Let $\delta x-t \geq 0$, since in this case, $\frac{0.25 \delta^2 x^2}{\delta x - t} \geq t$ then optimum value is $f(x,y) t/(t+1) = t$ with optimum solution $x = \frac{2t}{\delta}$. Now we need to workout the case $\delta x - t <0 \implies x < t$. So now the final solution is $x = \frac{2t}{\delta}$ under the condition $x \geq t$. Under this condition, if $x = \frac{2t}{\delta} > x_1$, by choosing $\rho_2 = -1$ i.e., $\delta = 2 \sqrt{t/x}$, we get $x = \frac{2t}{\delta} = \sqrt{tx} > x_1 \implies t > x$. Hence under the condition $x \geq t$, the solution is $x =\frac{2t}{\delta}$.
You need to work out the case $x_1 < t$.