Let $S$ be the unit sphere in the first octant that is closest to the origin, and let $T$ be the largest sphere in the first octant that intersects $S$ in just one point. I'm trying to determine equations for $S$ and $T$, without using any calculus.
By symmetry, I think $S$ must have equation
$$(x-1)^2+(y-1)^2+(z-1)^2=1\tag{1}$$
But what about for $T$? We need to find $(x_0, y_0, z_0)$ and $R$ such that
$$(x-x_0)^2+(y-y_0)^2+(z-z_0)^2=R^2\tag{2}$$
is contained entirely in the first octant and hits $S$ in just one point.
That means we want the system of equations (1) and (2) to have just one solution. Subtracting equation (1) from equation (2) gives the plane
$$2x(1-x_0)+2y(1-y_0)+2z(1-z_0)=2+R^2-(x_0^2+y_0^2+z_0^2)$$
If we assume $x_0=y_0=z_0=R$ by symmetry (so that $T$ is tangent to the coordinate planes), we get
$$2x+2y+2z=2\frac{1-R^2}{1-R}=2(1+R)$$
so
$$x+y+z=R+1$$
We want to choose $R$ so that this plane intersects (1) and (2) in just one point. I'm not sure how to proceed from here.
If I further assume that the three coordinates where the spheres touch are all equal, i.e. $x=y=z$, the plane equation says
$$3x=R+1\implies x=\frac{R+1}{3}$$ and substituting this into (1) gives $$3(\frac{R}{3}-\frac{2}{3})^2=1\implies \frac{R}{3}-\frac{2}{3}=\frac{\sqrt{3}}{3}\implies R=2+\sqrt{3}$$
This looks right when I plot it, but I'm not sure how to rigorously justify it.
If a sphere of radius $r$ touches the coordinate planes then the distance from the origin to its center is $r \sqrt{3}$. The farthest point on the sphere from the origin is at distance $r\sqrt{3}+ r$, and the closest at $r\sqrt{3} - r$. Say now you have a smaller sphere of radius $\rho$ that touches the three coordinate planes and the previous one. Say this new one is of radius $\rho$. The farthest point from the origin on the small sphere is the closest to the origin on the bigger sphere. So $$\rho \sqrt{3} + \rho = r\sqrt{3} - r$$ Get $$\rho = \frac{\sqrt{3}-1}{\sqrt{3}+1} \cdot r= (2 - \sqrt{3})r = \frac{r}{2+ \sqrt{3}}$$