Original post is edited after getting some suggestions;
I am looking for a fast algorithm which is able to detect outer most boundary of a plane for given set of points. Suppose, I have 3D point clouds and points are segmented as belonging to different (identified) planes. Now I want to extract outer most points of each plane. The problem can be considered as a 2D case by projecting x,y coordinates of each point to the XY Plane. So what I am expecting is fast, precise algorithm, which is able to detect all the boundary points along very irregular borders. convex hull doesnt fit for me as it fails on irregular cases.
•publications relavent to this, codes and psudo codes are expecting to implement in c++. thank you
I'm not sure about your problem statement, but you might find the Hough transform, (in 3D) usefule. For example: http://plum.eecs.jacobs-university.de/download/3dresearch2011.pdf
Added: statement misunderstood, it seems that the planes are already identified, we just want to find out the borders of the figures that each subset of points determine over each plane. For that, I'd project the points to the respective plane and apply iterative algorithm, starting with the convex hull and deleting/splitting borders. For example: http://forja.uji.es/docman/view.php/43/83/border_cloud_points.pdf