Start with an isosceles triangle with a vertex angle $\alpha>\frac{\pi}{2}$ and draw two similar triangles on its sides, continue ad infinum. Each triangle will have sides $\frac{1}{2 \sin \alpha/2}$ smaller, or the area $\frac{1}{4 \sin^2 \alpha/2}$ smaller. For $\alpha>\frac{\pi}{2}$ the final area of the fractal will be finite.
In the general case the boundary of the resulting figure will have a fractal shape. However, I'm interested in the borderline case of:
$$\alpha=\frac{2 \pi}{3}$$
Then for the first 2 steps we obtain:
And in fact every "groove" like the central one will be filled completely by the smaller triangles, because we have:
$$S_n=\frac{1}{3} S_{n-1}$$
As $2$ new triangles are created at each step, we have:
$$S_0 \sum_{n=1}^\infty \frac{2^n}{3^n}=2S_0$$
One half of this area ($=S_0$) will be spent to fill the groove, the other will stay "outside". (This is wrong see the paragrath in the end. Actually the everything goes inside the groove, since half of the triangles overlap the older ones).
As I ask in the title: what is the final shape of the boundary of this fractal going to be? Will it have a smooth $1D$ boundary or will it stay fractal?
Unfortunately, I haven't been able to just program this fractal yet, but also I would like to hear a theoretical answer if it's possible to determine without doing the experiment.
I have started to figure out how to build the program in cartesian coordinates. Basically we have a one dimensional array of points (collected over all previous iterations) and between each neighbour points we build a perpendicular with the length $l_n=l_{n-1}/ \sqrt{3}$. Then we add this new point to our array exactly between the two points, and continue further down the array.
If two initial points have the coordinates $(x_1,y_1)$ and $(x_2,y_2)$ then the new point will be:
$$x=\frac{x_1+x_2}{2} \pm \frac{y_1-y_2}{2 \sqrt{3}}$$
$$y=\frac{y_1+y_2}{2} \pm \frac{x_1-x_2}{2 \sqrt{3}}$$
The choice of signs is not always easy, I have not determined the universal rule yet. Here's the result of 4 full iterations and the start of the 5th, made by hand:
An important correction based on the comment below: the figure eventually starts to overlap itself. But because the triangles add up perfectly to fill the "grooves", let's just add a rule to stop constructing new triangles in places the old ones touch each other with their sides.
I used LibreOffice Draw to get more iterations in a crude way (by copying and scaling the whole previous pattern) and I got this at just the 3rd step. Looks like a fractal boundary after all (though the impression might be wrong, since more iterations should cover all the "grooves").
Because of the overlap reveal I have also added a second question: what is the area of the fractal if we don't count the triangles which overlap the previous ones?
This is not a complete answer but I'm certain that it contains some relevant details. I wish I had a bit more time to think about this interesting problem.
First of all, your original self-similar set can be generated using an iterated function system scheme with functions
\begin{align*} f_0(x,y) &= \left( \begin{array}{cc} \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \frac{1}{2} & \frac{\sqrt{3}}{2} \\ \end{array} \right)\left( \begin{array}{c} x \\ y \\ \end{array} \right) \\ f_1(x,y) &= \left( \begin{array}{cc} \frac{\sqrt{3}}{2} & \frac{1}{2} \\ -\frac{1}{2} & \frac{\sqrt{3}}{2} \\ \end{array} \right)\left( \begin{array}{c} x \\ y \\ \end{array} \right) + \left( \begin{array}{c} \frac{1}{2} \\ \frac{1}{2 \sqrt{3}} \\ \end{array} \right) \end{align*}
An image of this set looks like so:
You can generate images of self-similar sets using the app on this webpage. Note that the webpage has an "examples" menu and the last example (labeled "Stack example") is this very example.
These types of images can be generated using the IFS scheme. That is, we plot the collection of sets of the form $$S_{i_1,i_2,\ldots,i_k} = f_{i_1}\circ f_{i_2}\circ\cdots\circ f_{i_k}(S),$$ where $S$ is an initial seed set. For example, if $k=5$ so that we want to generate an approximation of level 5 and $S$ is the unit interval on the $x$-axis in the plane, we get an image like so:
Note the segment circled in red. That segment is exactly where two functions in the level 5 IFS clash. That is:
$$f_1\circ f_0\circ f_0\circ f_0\circ f_0(S) = f_0\circ f_1\circ f_1\circ f_1\circ f_1\circ f_1(S).$$
If we would like to generate your "boundary" set, then we don't want sequences of zeros and ones that start with $(0,0,0,0,1)$ or $(1,1,1,1,0)$. By self-similarity, we probably don't want those sequences to appear anywhere in the address of a point.
But there's more. The circled segment forms the base of a Y; the two branches of the Y will clash in two more steps. As a result, we also want to exclude sequences of the form $(1, 1, 0, 1, 1, 1, 0)$ or $(0, 0, 1, 0, 0, 0, 1)$.
Taking this much into account, I was able to generate the following image:
This is generated as follows:
A closer examination shows there's more to exclude but, again, I'm getting a bit tired.
I should mention that there is a history of examining interesting subsets of self-similar sets using exactly this type of restriction. For a couple of examples, see