I get a bunch of not really random points, which should be abled to formed a single line, the example looks like the following:
In this sample, we could easily imagine there is a line formed by the center of these circles, which looks like the following picture:
I would like to know if there are some algorithm that I could use to find out the vector (or coordinate of start point and end point) of of the black line that I draw at second graph.
The standard solution is least-squares fitting of a straight line. You find a line such that the total squared distance from the given points to the line is minimum.
If you search the Web, you will find many solutions based on the vertical distance (formulas are simple). I recommend to use the perpendicular distance, which is more isotropic. It is also called total least-squares.
Actually, this is an instance of the Principal Component Analysis problem where you find a linear trend in a point cloud. You achieve this by 1) computing the center (average) of the cloud, 2) computing the main directions of the cloud by solving an Eigenproblem.
The math may look scary, but they are not in the 2D case. (Unfortunately it is hard to find a web page just showing the 2D case and you have to swallow all the linear algebra involved.)
This is not the only consideration: depending on your data set, you may have outlier points (i.e. alien points which actually do not belong to the line but are present in your data set by some accident). They will strongly influence the least-squares fitting and must be discarded. For this, you can resort to the so-called robust fitting methods, for instance by minimizing the total distance rather than the total squared distance.
A simple yet powerful approach is RANSAC: take points in pairs and compute the sum of distances from the line they form to all other points. Keep the pair that minimizes the error. Usually the number of pairs is excessive ($N(N-1)/2$) and you try a certain number of pairs randomly. If $N$ is small, you can afford exhaustive attempts.
UPDATE: the total least-squares solution.
Assume the data points have been centered, and let the line have angle $t$. The sum of the squared distances is $$S(t)=\sum_{i=1}^n(x_i\sin t-y_i\cos t)^2=\sin^2t\sum_{i=1}^nx_i^2-2\sin t\cos t\sum_{i=1}^nx_iy_i+\cos^2t\sum_{i=1}^ny_i^2\\=S_{x^2}\frac{1-\cos2t}2-S_{xy}\sin2t+S_{y^2}\frac{1+\cos2t}2.$$ This sum is minimized when the first derivative cancels: $$(S_{x^2}-S_{y^2})\sin2t-2S_{xy}\cos2t=0.$$ Solve for $2t$, take the solution that is a minimum, and undo the centering.