Draft image made in paint.
I wanted to figure out the dimensions of an equiangular hexagon that fits inside a square and has this specific shape, simply because I wanted to turn it into vector art with inkscape.
The things I do with my free time, eh?
While doing the math I ran into an issue where I couldn't mathematically prove the lengths of a and b as a function of the square side length, thus I am stuck at an equality where $$ a+b = \frac{\sqrt{3}}{2} \bigl(a+b\bigr) $$ which obviously isn't very helpful.
Anyone got ideas on how I could get out of this mess?
Your measurements and geometry are good. The smallest rectangle that encloses the hexagon has a height that is $\tfrac{\sqrt{3}}{2}$ times the width. So, if you want the hexagon (with this orientation) to fit inside a square, then you're limited by the longer dimension: the width.
Say your square has side length $s$. If you want the hexagon with dimensions $a$ and $b$ to be as large as possible (fit snuggly in the square), then $$ s = \tfrac{\sqrt{3}}{2} \bigl( a + b \bigr) $$ so given a square of side lenth $s$, you need $$ a + b = \frac{2}{\sqrt{3}} \cdot s $$ It's interesting that the sum of the two side lengths of the hexagon is the quantity that's constrained, but beyond that, there is no further constraint, other than the fact that $a, b \geq 0$.
The extreme cases where $a = 0$ so $b = \tfrac{2}{\sqrt{3}} \cdot s$ and vice versa are interesting, as they aren't really hexagons at all, but rather equilateral triangles! In any case, either extreme polygon fits in the given square, as do all the hexagons in between.