I want to show that $\lim_{(x,y)\rightarrow (1,1)}(x^2+y^2) \neq 3$.
My try:
$$ |x^2+y^2 -3| = |(x-1)^2 + (y-1)^2 +2x+2y-1-1 - 3| \leq |(x-1)^2+(y-1)^2|+|2x+2y-5| < \epsilon$$
$$\sqrt{(x-1)^2+(y-1)^2}^2 +|2x+2y-5|<\epsilon $$
Because we know that: $\sqrt{(x-1)^2+(y-1)^2}< \delta$ we can do the following:
$$\sqrt{(x-1)^2+(y-1)^2}^2 +|2x+2y-5|< \delta^2 +|2x+2y-5| < \epsilon $$
Now let's take a look of delta again: $(x-1)^2+(y-1)^2 = x^2+1-2x+y^2+1-2y = x^2+y^2-2x-2y+2<\delta^2$
$x^2+y^2-\delta^2 < 2x+2y - 2$
Can we use this to do the following:
$$\delta^2+|2x+2y-2|+|-3|<\delta^2+|x^2+y^2-\delta^2| < \delta^2+|x^2+y^2|+|-\delta|< \epsilon$$
$$2 \delta^2 + |x^2+y^2| < \epsilon $$
And then can we repeat again for $|x^2+y^2|$ and we can just generate sums of $\delta$ and we get a series that diverges because $\delta > 0$ and thus $\epsilon$ cannot satisfy the inequality.
Is the idea of the proof correct.
How to do this proof the correct way?
What you've written is mostly a lot of inequalities that spin around in circles without going anywhere.
Instead, before diving into inequalities, you should take a moment first to work through the logic of the problem, which will tell you the primary goal of your proof.
You want to prove that the following equation is false: $$\lim_{(x,y) \to (1,1)} x^2 +y^2 = 3 $$ Applying the definition of a limit, you want to prove that the following statement is false:
By applying the rules of negation, you are trying to prove that the following statement is true:
This tells you that the primary goal of your proof is to first find an appropriate value of $\epsilon>0$, which you will then use to prove the statement "For all $\delta > 0$ there exists $(x,y) \in \mathbb R^2 - \{(1,1)\}$ such that $|(x-1,y-1)| < \delta$ and $|x^2+y^2-3| \ge \epsilon$."
A little bit of intuition should guide you to an appropriate value of $\epsilon$. Draw a picture in the number line: we know that (as $(x,y)$ approaches $(1,1)$) the value of $(x^2+y^2)$ approaches $2$, which differs from $3$ by $|3-2|=1$. Cut that in half and we get $\epsilon = \frac{1}{2}$. A triangle inequality calculation, using the equation $|3-2|=1$, tells us that the following implication is true for all $(x,y) \in \mathbb R^2$:
Now we have a concrete goal: For all $\delta>0$, find an appropriate $(x,y) \in \mathbb R^2 - \{(1,1)\}$ and use it prove that $|x-1,y-1| < \delta$ and $|x^2+y^2-2| < \frac{1}{2}$. Once you've done that, you may immediately conclude that $|x^2+y^2-3| > \frac{1}{2}$, and the proof is complete.
Now there's one final thing to determine: What conditions on $(x,y)$ imply $|x^2+y^2-2| < \frac{1}{2}$? Well, that's what your knowledge of continuity should supply: you know that there exists $\delta_1 > 0$ such that $|(x-1,y-1)| < \delta_1$ implies $|x^2+y^2-2| < \frac{1}{2}$; I'm confident you could actually find $\delta_1$ explicitly.
And now for the final logic of the proof: for any $\delta > 0$, choose $(x,y) \in \mathbb R^2 - \{(1,1)\}$ such that $(x-1,y-1) < \min\{\delta,\delta_1\}$; then follow through the implications already proved, to conclude that $|x^2+y^2-3| > \frac{1}{2}$.