In one of my applications, I need to represent a line under 3D polar coordinates system. In 2D, we can define a line by a distance to the origin and then a angle indicating the direction of the line (see the following example).
However, I get a little confused under 3D case. It seems that we need 3 parameters to define the orientation of the line. Look at the following example, with (r, $\varphi$, $\theta$) it defines only a 3D plane. Any idea of how to put the third angle parameter, say $\omega$. Is there a standard way to put $\omega$?
This might also be known as Hough transformation. Correct me if I am wrong about this.
Three numbers is not enough to represent a 3D (infinite) line. You need at least 4 numbers, and using 5 or 6 makes life easier. The obvious representation with 6 numbers is a point (3 numbers) and a vector (3 more numbers). To get down to 5, you use a unit vector, which can be represented by two numbers (e.g. spherical polar angles).
To use the technique you're suggesting, you will need to use one more number (in addition to $r$, $\theta$, $\phi$). Let's call this additional number $\psi$.
Imagine you start with a vertical line, parallel to the $z$-axis, through the point $(r,0,0)$ on the $x$-axis. Rotate this line around the $z$-axis by an angle $\theta$ and then rotate upwards by an angle $\phi$. Your line will now lie in the shaded plane shown in your second picture, but it's direction might not be correct. Use the angle $\psi$ to rotate the line from its current position to the correct one.
The angles $\theta$, $\phi$, $\psi$ are often called Euler angles. Three Euler angles can be used to represent any 3D rotation. They are horribly confusing because there are many different ways to define them (depending upon the axes chosen for rotation, and order of rotations). For some details, see this Wikipedia page.