I want to fit a 3D line to a 3D line point cloud using numerical optimization. Currently, I'm using Steepest Descent and the error function is a function of $\alpha$, $\beta$, $\gamma$; that is, 3 angles defining the fitting line.
This raises the question of how to validate the result. I want to visualize the function so that I can check the objective function's minima & maxima. Can I do this with MATLAB and should I? Is this a common approach to analyze an objective function when there is no math formation for it?
So, "3d line pointcloud" is apparently a set of points $P_j \in {\mathbb R}^3$ that are close to a line. I'm not sure what you mean by "3 angles defining the fitting line". A straight line in ${\mathbb R}^3$ in general has 4 degrees of freedom, not 3. Three angles determine a rotation, not at all the same thing.
But what I would do is find the average $\overline{P}$ of your points and an eigenvector $V$ of the covariance matrix for its largest eigenvalue. The line can then be represented as $\overline{P} + t V$.