I came across this definition
but it's confusing to me.
Shouldn't $A$ be $(x_1,y_1)$ instead of $(x_1,x_2)$? Same for $Y$. And even if $A$ was correct like that then shouldn't the formula for the distance be written as $$d(X,Y) =\sqrt{(y_1-x_1)^2+(y_2-x_2)^2}$$
What am I getting wrong?

Indeed there is a typo and your expression
$$d(X,Y) =\sqrt{(y_1-x_1)^2+(y_2-x_2)^2}$$
is the correct one by direct application of pythagorean theorem.