I have two segments in 2D space, defined by their endpoint x and y coordinates. How can I find a best-fit line using vector algebra (formally, that minimizes the integral of square-distance from it to every bit of the segments)? Is such a line unique?

How do you define a "best fit line"... that is the question.
Maybe a solution is to find $a,b \in \mathbb{R}$ such that if the line $(L)$ is defined by $y=ax+b$ then $\sum_{i=1}^4 d^2(L,P_i)$ is minimum where $P_1, \dots, P_4$ are the end points of the two segments. That is what you suggest by minimizing the integral of square-distance.