Math is not my first skill, so I figured it might be useful to ask mathematicians. I?m using a clustering algorithm to group 2D segments using M and Q in order to find coherent lines. For the sake of escaping infinity I compute the arctan of M (actually I do it also for Q) so I have a values between -$\frac{\pi}{2}$ and $\frac{\pi}{2}$.
What I would like is to find a way to represent the M such as values near -$\frac{\pi}{2}$ is clustered with values very close to $\frac{\pi}{2}$.
No need to say that just apply an offset to the rotation just shift the problem.
I'm currently using hdbscan as clustering algorithm.
No sure how to tag it
You can use the pair $(\cos 2\alpha,\sin 2\alpha)$ to code $\alpha=\arctan M$.
EDIT: The mention of a custom distance function made me think of computing a custom distance straight from the coordinates of the endpoints, rather than dealing with slopes and intercepts. One possibility that crossed my mind is to calculate how far the endpoints are from the line that passes through the midpoints of the segments. You can do that easily enough with the cross product.
Let $(x_1,y_1)$ and $(x_2,y_2)$ be the endpoints of one segment, $(x_3,y_3)$ and $(x_4,y_4)$ those of another segment. Let $(u,v)$ be the vector from the midpoint of first segment to the midpoint of the other one, that is, $$\begin{eqnarray*} u&=&\frac{x_3+x_4}{2}-\frac{x_1+x_2}{2},\\ v&=&\frac{y_3+y_4}{2}-\frac{y_1+y_2}{2}.\\ \end{eqnarray*} $$ Now the distance of $(x_1,y_1)$ from the line passing through the midpoints is $$\frac{|(x_1-x_2,y_1-y_2)\times(u,v)|}{2\sqrt{u^2+v^2}} =\frac{|(x_1-x_2)v-(y_1-y_2)u|}{2\sqrt{u^2+v^2}}. $$ Naturally, the point $(x_2,y_2)$ is at the same distance from the line, on the other side of it. Similarly, the distance of the endpoints of the other segment from the line is $$\frac{|(x_3-x_4,y_3-y_4)\times(u,v)|}{2\sqrt{u^2+v^2}}. $$ I suggest using the sum of the squares of the above distances, scaled to remove the constant factor $1/2$, as your custom distance function for the clustering algorithm. The expressions simplify a bit when you square them: $$ \frac{((x_1-x_2)v-(y_1-y_2)u)^2+((x_3-x_4)v-(y_3-y_4)u)^2}{u^2+v^2}. $$