Let $\hat{\mathbf{x}}, \hat{\mathbf{y}},\hat{\mathbf{z}}$ be the unit vectors defining my 'world' coordinates.
Consider the following:
In this example, $$\mathbf{r}_1 = r_{1,x} \hat{\mathbf{x}} + r_{1,y} \hat{\mathbf{y}} + r_{1,z} \hat{\mathbf{z}} $$
and the equation for the plane is, for a normal vector $\mathbf{n} = (n_1,n_2,n_3)$,
$$n_1 (x-r_{1,x})+n_2 (y-r_{1,y}) + n_3 (z-r_{1,z})=0$$
I've defined the plane where this circle lives.
The problem is, I don't want an equation of a plane, but instead a transformation of my initial coordiante system into a new one.
I want to define a coordinate system $\hat{\mathbf{x}}', \hat{\mathbf{y}}',\hat{\mathbf{z}}'$ at $\mathbf{r}_1$ that's rotated such that,
$$n_1 (x-r_{1,x})+n_2 (z-r_{1,y}) + n_3 (z-r_{1,z})=0$$
lives in the $\hat{\mathbf{x}}', \hat{\mathbf{y}}'$ plane. That way, I can use the $\hat{\mathbf{x}}', \hat{\mathbf{y}}'$ plane to define a circle but can treat it like a standard 2D plane with polar coordinates.
How do I do this? I'm particularly confused given that I gather rotation and translation aren't necessarily commutative, so I'm confused how to do this.
First step is to find a basis of the plane you are interested in. First find any vector $\mathbf{v}$ in your plane, which can be anything orthogonal to $\mathbf{n} = \left(\begin{array}{c}n_1 \\ n_2 \\ n_3 \end{array} \right)$. You should choose it to have length 1 to make things easier. We will use this as your new $\hat{\mathbf{x}}'$, which will point along the $x$-axis in your new reference plane.
To find your $\hat{\mathbf{y}}'$, simply take the cross product $\mathbf{v} \times \mathbf{n}$. This is guaranteed to be orthogonal to both $\mathbf{n}$ and $\mathbf{v}$ by standard properties of the cross product. Being orthogonal to $\mathbf{n}$ means it also lives in your plane. You should again rescale so it has length 1.
Then $\hat{\mathbf{z}}'$ will just be $\mathbf{n}$ itself (or rather the unit vector pointing in the direction of $\mathbf{n}$), since it already points "up" relative to your plane.
Now form the matrix $A$ whose columns are $\hat{\mathbf{x}}'$, $\hat{\mathbf{y}}'$, and $\hat{\mathbf{z}}'$. This matrix will send $\hat{\mathbf{x}}$ to $\hat{\mathbf{x}}'$, and so on... This sends the $x,y$ plane to the plane through the origin that is parallel to your desired plane. Finally, you simply add $\mathbf{r}$ to translate everything away from the origin.
Summary: if you have a vector $\mathbf{a} = \left(\begin{array}{c} x \\ y \\ 0 \end{array} \right)$ in the $x,y$-plane, then the corresponding point in your new plane will be $A \, \mathbf{a} + \mathbf{r}$.