There is a puzzle in Recreational Math in Khan Academy which is very difficult to solve. This puzzle involves using a restricted amount of transformations. The question does not appear to work in the intended way, as the centre of dilation is at (5, 5), when the question describes that it should be at (5, -5), but it should still be quite possible to solve. Here is a screenshot of the question:
You can use rotations of a multiple of 15˚ around the point (-5, -5), dilations around (5, 5) of a multiple or divisor of two and reflections on the line: $y = -x - 5$. is there a good method or strategy to finding the answer, or are your best chances to be familiar with how the transformations work, and mess around brute force style until you find the answer.

If the dilation center is at $(5,5)$ as stated in the question text, rather than the $(5,-5)$ as stated in the graphic, then the following sequence does the trick:
This was found by a combination of manual messing around (after drawing things in a simplified coordinate system where the reflection line is an axis and the other important points have small integer coordinates) and theoretical knowledge.
First: By manual messing around (and several false starts, including an orientation mistake that was part of this answer for hours) I found this 5-transformation sequence:
Then: I noticed that steps 1-2 and 4-5 were each other's inverses, so in group theoretic what I had here was a conjugation of the 90° rotation. Effectively what needs to be done is to rotate by 90° around the point $(2\frac12,2\frac12)$ and what 1-2 in the 5-step sequence do is just move $(2\frac12,2\frac12)$ to the rotation center at $(-5,-5)$. Once I realized that, it is clear that there is an easier way to do this: just scale by 4 around $(5,5)$.
This insight also allows us to produce a 7-step solution for the variant where the dilation point is $(5,-5)$:
It is interesting that steps 1-3 here are themselves a conjugation of the reflection -- blowing up by a factor of 2 transforms the line $x+y=-2\frac12$ (in which we can reflect $(2\frac12,2\frac12)$ to move it to $(-5,-5)$) into the line $y=-5-x$ that we can reflect things in.
The above solutions shows that some theoretical knowledge is useful when thinking about the problem. However, I don't think there's a any nice and systematic way to solve this kind of puzzles based on deep properties of the transformations. Of course one could phrase it as a graph shortest-path problem in a graph where each vertex is a particular transformation of the plane (and there are exactly two similarity transformations that take the starting polygon to the ending one), but that is a very brute force approach.
One additional insight I did find useful for the initial messing-around step is that as long as the dilation factors are always rational, it is not useful to rotate by angles other than multiples of 90° -- doing so would lead to irrational coordinates that can't easily be made rational by subsequent transformations.