Let $P$ be the point in $XY$ plane and $P'$ is a point such that $OP.OP'=r^2$, where $O,P,P'$ are collinear and $O$ divides $P$ and $P'$ externally and $O$ is the origin.
If the point $P$ lies on the line $x+y=1$, then $P'$ traces
(A) A straight line of slope $\frac{1}{r}$
(B) A circle of radius $\frac{r}{\sqrt{2}}$
(C) A circle of radius $\frac{r^2}{\sqrt{2}}$
(D) A circle of radius $\frac{r^2}{2}$
I used the following approach
If a tangent and secant segment are drawn to a circle from a exterior point then the square of the measure of the secant segment and its external secant segment are illustrated as
$UV^2=UX.UY$
Same concept if applied here where $P$ and $P'$ are collinear with the orgin.
Suppose the locus of P is a straight line $x+y=1$ and let the required circle be $x^2+y^2+2gx+2fy+c=0$
$c=r^2$ by property not able to proceed as we we need to trace locus of $P'$.
But we know that $(h+g)^2+(k+f)^2=(\frac{-g-f-1}{\sqrt{2}})^2$
Answer (C). Here is why.
Your question deals with transformation called (geometrical) inversion (more precisely inversion with pole $0$ and power $r$) but we do not assume knowledge of it in the solution below. See figure at the bottom.
Let us work with polar equations :
If $\vec{OM}=\binom{x}{y}=\rho \binom{\cos \theta}{\sin \theta}$, plugging $x=\rho \cos \theta, y=\rho \sin \theta$ into the equation of straight line $x+y=1$ gives its polar equation
$$\rho = \rho_1=\frac{1}{\cos(\theta)+\sin(\theta)}$$
Therefore, the polar equation of the image of this line by the inversion is such that $\rho_1\rho_2=r^2$, giving :
$$\rho = \rho_2=\frac{r^2}{\rho_1}=r^2(\cos(\theta)+\sin(\theta))=r^2 \sqrt{2}\cos(\pi/4-\theta)\tag{1}$$
which is the polar equation of a circle passing through the origin with diameter $r^2 \sqrt{2}$, i.e., with radius $\frac{r^2}{\sqrt{2}}.$
Fig. 1 : Case $r=2$. The image of point $A$ on the (blue) straight line is point $A'$ on the (red) circle. The dotted circle is the set of invariant points, i.e., points that are unchanged by the transformation. I have positionned a generic point $M$ with polar angle $\theta$ in order to understand through an easy computation in right triangle with leg $OM$ how equation (1) can be retrieved.