Consider a circle with equation $x^2+y^2=1$ over $\mathbb{R}$ and hyperbola $xy=1$. Then with homogenenous coordinates we have that the circle has equation $X^2-Y^2-Z^2=0$ which gives the matrix $$A=\begin{bmatrix}1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & -1\end{bmatrix}.$$
For the hyperbola we have the equation $X^2-XY=1$ so we homogenize the matrix to look like $$B=\begin{bmatrix}2 & 0 & 0 \\ 0 & 0 & -1 \\ 0 & -1 & 0\end{bmatrix}.$$
I would like to map the circle to the hyperbola with a matrix $M$ such that $M^{-1}AM^{-T}=B$ where $-T$ denotes the inverse transpose of $M$. The problem is I am not sure how to get $M$. If we take the base points $(1:0:0),(0:1:0),(0:0:1)$ , only two of these belong to the circle over $\mathbb{R}$ and the circle has no points at infinity. Any hint is appreciated.
First of all, the equation of the circle is $X^2+Y^2-Z^2=0$ correctly, so $$A=\begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1\end{bmatrix}.$$ Also, the equation of the hyperbola is $$XY-Z^2=0,$$ so $$B=\begin{bmatrix}0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & -2\end{bmatrix}.$$ Now we will find the projective transformation that sends the hyperbola to the circle. We will use that a conic is determined by its three points and the tangents at two of these points. So we will send three points and two corresponding tangents of the hyperbola to points and tangents of the circle.
First, $[1,0,0]$ is on the hyperbola, let its image be $[1,0,1]$.
$[0,1,0]$ is also on the hyperbola, let its image be $[0,1,1]$.
The tangents of the hyperbola at these points are the $x$-axis and $y$-axis, their intersection is $[0,0,1]$. Let its image be $[1,1,1]$, because the tangents of the circle at $(1,0)$ and $(0,1)$ meet at $(1,1)$.
Finally, $[1,1,1]$ is on the hyperbola, let its image be an arbitrary point of the circle, for example $[-1,0,1]$.
We use that in the coloumns of the matrix of a linear transformation we find the images of the base vectors. Thus our desired matrix is of form $$M=\begin{bmatrix}a & 0 & c \\ 0 & b & c \\ a & b & c\end{bmatrix}.$$ (For example, the first coloumn is $[a,0,a]$, because the image of $[1,0,0]$ is $[1,0,1]$, which is the same point as $[a,0,a]$ for any $a\neq 0$.)
Since the image of $[1,1,1]$ is $[-1,0,1]$, we get the equation $$\begin{bmatrix}a & 0 & c \\ 0 & b & c \\ a & b & c\end{bmatrix}\begin{bmatrix}1 \\ 1 \\ 1\end{bmatrix}=\begin{bmatrix}-1 \\ 0 \\ 1\end{bmatrix}.$$ Solving this equation we get $a=1$, $b=2$, $c=-2$. Thus the matrix of the projective transformation that sends the hyperbola to the circle is $$M=\begin{bmatrix}1 & 0 & -2 \\ 0 & 2 & -2 \\ 1 & 2 & -2\end{bmatrix}.$$ You can check that if $N$ is the inverse of this matrix, then $NAN^T=B$ (up to some factor).