I am wondering if anyone knows how to compute the direction of a cylinder by using the 10 coefficients.
For example, we have the equation of a cylinder as
$$c_0+c_1x+c_2y+c_3z+c_4x^2+c_5y^2+c_6z^2+c_7xy+c_8xz+c_9yz = 0$$
How can we get the axis of the cylinder?
Thank you.
Gradient
The direction of the cylinder is the direction where the value of the left hand side does not change. So $(a,b,c)$ is the direction if
$$\forall x,y,z\in\mathbb R:f(x,y,z)=f(x+a,y+b,z+c)$$
where $f(x,y,z)$ denotes the left hand side of your equation. The change of $f$ in a given direction can also be described using the gradient $\nabla f$. What you want is a direction which is orthogonal to that gradient at any point in space. So what is the gradient?
$$\nabla f=\begin{pmatrix} \frac{\partial\,f}{\partial x} \\ \frac{\partial\,f}{\partial y} \\ \frac{\partial\,f}{\partial z} \end{pmatrix} = \begin{pmatrix} c_1 + 2c_4x + c_7y + c_8z \\ c_2 + c_7x + 2c_5y + c_9z \\ c_3 + c_8x + c_9y + 2c_6z \end{pmatrix} = \begin{pmatrix} c_1 & 2c_4 & c_7 & c_8 \\ c_2 & c_7 & 2c_5 & c_9 \\ c_3 & c_8 & c_9 & 2c_6 \end{pmatrix}\cdot\begin{pmatrix} 1 \\ x \\ y \\ z \end{pmatrix} $$
To find a vector orthogonal to this vector field, you want to find $(a,b,c)$ in such a way that its scalar product with $\nabla f$ is zero for all possible choices of $(x,y,z)$. You can think of that scalar product as the product between a row vector $(a,b,c)$ and the matrix-times-column-vector product from the notation above. If the resulting scalar is always zero, no matter the vector to the right, that means that the left two factors of the product must already be zero. So you have the equation
$$ (a,b,c)\cdot \begin{pmatrix} c_1 & 2c_4 & c_7 & c_8 \\ c_2 & c_7 & 2c_5 & c_9 \\ c_3 & c_8 & c_9 & 2c_6 \end{pmatrix} =(0,0,0,0) $$
You might be more comfortable if you transpose this system:
$$ \begin{pmatrix} c_1 & c_2 & c_3 \\ 2c_4 & c_7 & c_8 \\ c_7 & 2c_5 & c_9 \\ c_8 & c_9 & 2c_6 \end{pmatrix} \cdot\begin{pmatrix}a\\b\\c\end{pmatrix}=\begin{pmatrix}0\\0\\0\\0\end{pmatrix} $$
In general this is an overdetermined linear system of equations. But that's because the general form of a quadric surface is not restricted to cylinders. If the object describe by your coefficients is indeed a cylinder, then the above system of equations will have a one-dimensional solution space. Which means that the matrix will only have rank $2$. In essence it would be enough to only consider two of the four equations, as long as you are careful in picking them.
Eigenvectors
Here is an alternative: start by writing your quadric as
$$(x,y,z,1)\cdot\begin{pmatrix} 2 c_{4} & c_{7} & c_{8} & c_{1} \\ c_{7} & 2 c_{5} & c_{9} & c_{2} \\ c_{8} & c_{9} & 2 c_{6} & c_{3} \\ c_{1} & c_{2} & c_{3} & 2 c_{0} \end{pmatrix}\cdot\begin{pmatrix}x\\y\\z\\1\end{pmatrix}=0$$
Now compute the eigenvectors of the top-left $3\times 3$ submatrix. These form an orthogonal basis, and in that basis your quadric will be described by a diagonal matrix, which means that it will be axis-aligned. So you can easily pick the right coordinate axis from the set of eigenvectors. I guess it should be the one corresponding to an eigenvalue of zero, but please verify this before relying on it. How do you find an eigenvector for eigenvalue zero? You try to find a non-trivial (i.e. non-zero) solution to the following system of equations:
$$\begin{pmatrix} 2 c_{4} & c_{7} & c_{8} \\ c_{7} & 2 c_{5} & c_{9} \\ c_{8} & c_{9} & 2 c_{6} \end{pmatrix}\cdot\begin{pmatrix}a\\b\\c\end{pmatrix}=0$$
This is very similar to what I wrote above, except that some lines are reordered or scaled differently, and one line is omitted altogether.
See also this post on Stack Overflow for how to obtain the principal axes directions, size and rotation of a planar quadric.
Cross product
No matter which of the approaches above you follow, you will end up with a situation where you have to compute a vector $(a,b,c)$ which is orthogonal to some other vectors, while those other vectors will lie in a plane. Instead of solving a system of equations the hard way, you can make use of the cross product here, and compute e.g.
$$\begin{pmatrix}a\\b\\c\end{pmatrix}= \begin{pmatrix}2c_4\\c_7\\c_8\end{pmatrix}\times \begin{pmatrix}c_7\\2c_5\\c_9\end{pmatrix}$$
Make sure you choose two vectors for the right hand side which are neither zero nor multiples of one another.