I'm trying to find a better upper bound for two functions. How could I find it? I found these upper bounds.
$\textbf{First inequality:}$ If $(x,y) \in \mathbb{R}^{2}$ and $$f(x,y):=\frac{x^{2}(3y^{2}+x^{2})}{(x^{2}+y^{2})^{2}}$$ and let $x:=r\cos(\theta), y:=r\sin(\theta)$ and $r\not=0$, so we have that $$|f(x,y)|=\left| \frac{r^{4}(3\cos^{2}(\theta)\sin^{2}(\theta)+\cos^{2}(\theta)\sin^{2}(\theta))}{r^{4}}\right|=|\cos^{2}(\theta)\sin^{2}(\theta)(3+1)|=4|\cos^{2}(\theta)||\sin^{2}(\theta)|\leq 4(1)(1)=4$$
$\textbf{Second inequality:}$ If $(x,y) \in \mathbb{R}^{2}$ and $$g(x,y):=-\frac{2x^{3}y}{(x^{2}+y^{2})^{2}}$$ and let $x:=r\cos(\theta), y:=r\sin(\theta)$ and $r\not=0$, so we have that $$|g(x,y)|=|-1|\left| \frac{2r^{4}\cos^{2}(\theta)\sin^{2}(\theta)}{r^{4}}\right|=|2\cos^{2}(\theta)\sin^{2}(\theta)|\leq 2(1)(1)=2$$
I'm trying to find a better upper bound for this two functions. How could I find it? I found these upper bounds.
$\textbf{How could you find the best upper bound, for first and second inequality?}$
Your first upper bound $4$ does not occur.
By the way, by AM-GM $$\frac{x^2(x^2+3y^2)}{(x^2+y^2)^2}=2\cdot\frac{x^2\cdot\frac{x^2+3y^2}{2}}{(x^2+y^2)^2}\leq2\cdot\frac{\left(x^2+\frac{x^2+3y^2}{2}\right)^2}{(x^2+y^2)^2}=\frac{9}{8}.$$ The equality occurs for $x^2=\frac{x^2+3y^2}{2},$ which says that $\frac{9}{8}$ is a maximal value.
By the similar way we can get that: $$\max\frac{2x^3y}{(x^2+y^2)^2}=\frac{3\sqrt3}{8}.$$ Indeed, by AM-GM $$-\frac{2x^3y}{(x^2+y^2)^2}\leq\frac{2|x^3y|}{\left(3\cdot\left(\frac{x}{\sqrt3}\right)^2+y^2\right)^2}\leq\frac{2|x^3y|}{\left(4\sqrt[4]{\left(\frac{x}{\sqrt3}\right)^6y^2}\right)^2}=\frac{3\sqrt3}{8}.$$ The equality occurs for example for $x=-\sqrt3$ and $y=1$, which says that we got a maximal value.