Is it possible to represent a line in the general case using only two variables (namely
floats ordoubles)? By this I mean being able to convert a line into those two variables and successfully converting them back without loss of data or the algorithm failing for specific cases.
I know it can be done with three, since any line can be represented as the coefficients in Ax + By = C, but I can't represent a line using two using the coefficients in Ax + By = 1, since this representation produces infinite values for A or B if the line passes through the origin, and I can't use the coefficients m and b in y = mx+b since m becomes infinite for vertical lines.

@MBo's answer is quite optimal I think, unless you really don't want to deal with costly trigonometry, or if you don't want the problems with angle periodicity.
However, there is one more thing you can do. You can always squeeze infinities into finite range. Any monotonous function with two horizontal asymptotes (like arctan, tanh, or something like that) can be used to do that. When I'm dealing with positive values only, I like
$$a=\frac{A}{A+1}$$ and its inverse $$A=\frac{a}{1-a}$$
This transformation maps all positive reals including infinity to $[0,1]$. For $a\geq 0$ and $b\geq 0$, cross-multiply to remove fractions and write $$a(1-b)x+b(1-a)y=(1-a)(1-b)$$ which handles both infinite cases (when $a=1$ or $b=1$).
It is a bit more clumsy when you want to handle signed numbers: $$a=\frac{A}{|A|+1}$$ and the inverse: $$A=\frac{a}{1-|a|}$$
It follows $$a(1-|b|)x+b(1-|a|)y=(1-|a|)(1-|b|)$$
This represents all your lines symmetrically with numbers $-1\leq a,b\leq 1$, and requires one division per dimension for conversion, and only multiplication to get the equation.
As a bonus, you can also plot your lines on a unit square and see how the boundaries of the square represent the lines through the center, coordinate axes are vertical and horizontal lines, and the coordinate center is the line at infinity.
You need a special case for $a=b=\pm 1$: you can still represent them, but the equation is zero unless you treat the case specially and just fall back to $ax+by=0$. One
ifshould do it. Unfortunately, the mapping is still ill-conditioned around that point (very small values in the equation). You avoid that if you map to a circle instead of a square.It's always useful for programming to squeeze infinities into a finite range. It's useful for plotting infinite maps because then you never miss anything that gets out of range, while the center is not squeezed too much.
For mathematical purposes, it's better to preserve symmetry and map everything to the unit circle instead of square:
$$R=\sqrt{A^2+B^2}$$ $$a=\frac{A}{R+1}$$ $$b=\frac{B}{R+1}$$ and the inverses: $$r=\sqrt{a^2+b^2}$$ $$A=\frac{a}{1-r}$$ $$B=\frac{b}{1-r}$$
This avoids the corner problems and treats all the directions equally.