An equilateral triangle with side length $1$ unit contains three identical circles $C_1$, $C_2$ and $C_3$ of radius $r_1$, each touching two sides of the triangle. A fourth circle $C4$ of radius $r_2$ touches each of $C_1$, $C_2$ and $C_3$ as shown. Except for the contact points with $C_4$, none of the circles have any points in common with any of the other circles. Determine the values of $r_1$ and $r_2$ which minimize and maximize the sum $S$ of the areas of the four circles.
My attempt:
To maximize the area, I figured that I had to maximize the equation $S = 3\pi r_{1}^2 + \pi r_{2}^2$ by setting its derivative to zero and solving for $r_{1}$, then $r_{2}$. In order to change this formula so that I only had to work with $r_{1}$, I concluded after some experimentation that I could use $\frac{1}{\sqrt{3}} = 3r_{1}+r_{2}$ to substitute terms, where $\frac{1}{\sqrt{3}}$ is the distance between the vertex and the circumcenter. I got that $r_{2} = 3r_{1} - \frac{1}{\sqrt{3}}$. Following this process, I had $S = \pi(12r_{1}^2-2\sqrt{3}r_{1} + \frac{1}{3}), \frac{dS}{dr_{1}} = 24r_{1}-2\sqrt{3}$, and finally got a final answer of $r_{1} = \frac{\sqrt{3}}{12}$.
However, this result made my $r_{2}$ value negative, and then I got really stuck with what I was doing. I tried thinking it out, such as trying to figure out the domains and finding other equations to use, but I seem to be blanking out with this question. I'm now even more confused how I am supposed to find the values that result in the minimum $S$. Any help would be very appreciated!
It is true that $\frac{1}{\sqrt3} = 3r_1+r_2$, although you seem not to have a proof of this fact. But it does not follow that $r_2 \stackrel?= 3r_1 - \frac{1}{\sqrt3}.$ You have made a mistake in your algebra there, although once you square the expression for $r_2$ the mistake is canceled out.
A more serious mistake is when you thought that $r_1 = \frac{\sqrt{3}}{12}$ would maximize $\pi(12r_1^2- 2\sqrt3 r_1 + \frac13)$. Instead, it gives a minimum.
But the real trick here is to figure out how large $r_1$ can be before the figure can no longer satisfy the description. Here I would argue that the problem is ill-stated, because the configuration where the $r_1$ circles just touch each other is forbidden by the problem statement, yet it has a smaller area than any configuration in which the circles do not touch, and any of those configurations can be given less area by making the $r_1$ circles a little larger (just not quite large enough to touch). Hence no allowed configuration minimizes the area; it's like asking for the smallest value of $x$ such that $x > 1.$
On the other end, the problem does not say if the $r_2$ circle is required to be entirely contained in the triangle. If it is not, the area is maximized when $r_1 = 0,$ provided that you consider a circle of radius $0$ to be a circle. If the $r_2$ circle is limited to be inside the triangle then the minimum value of $r_1$ is larger than zero.
If I had to guess what the author meant, I would guess that they did not really mean to prevent the $r_1$ circles from touching, merely that they could not overlap, and I would guess that they meant the $r_2$ circle to be contained within (but possibly tangent to) the triangle. That gives you minimum and maximum values of $r_1.$ You will note that the value of $r_1$ that sets $dS/dr_1$ to zero is not in that range. You should know something about finding minima and maxima of a function when the domain is bounded, which you can then apply to this problem.