If I want to solve a generalized eigenvalue problem such as:
$$A x = \lambda x$$
The problem is to find eigenvectors corresponding to the largest eigenvalues (sometimes in an optimization problem that form it as a generalized eigenvalue problem).
I notice some people solve this problem by finding eigenvectors corresponding to the largest eigenvalues in absolute value.
Are these two method similar? What are the difference between them? I mean in what situation I should use eigenvalues in absolute value?
Finding the eigenvector with the largest magnitude eigenvalue is the easiest and cheapest, since you can use power iteration directly. See http://en.wikipedia.org/wiki/Power_iteration
If you need to find the eigenvector with largest eigenvalue, you must modify the procedure. Sometimes you know that the largest eigenvalue is also the largest-magnitude eigenvalue (for example, if your matrix is positive-definite) and so you can again use power iteration. Otherwise, you can find the largest-magnitude eigenvalue; if it is positive, you are done. Otherwise, construct a new spectrally-shifted matrix $M' = M - \lambda I$ where $\lambda$ is the largest-magnitude, negative, eigenvalue. The eigenvalues of $M'$ are now all non-negative, so find its largest-magnitude eigenvalue $\mu$ and corresponding eigenvector $v$; the largest eigenvalue of $M$ is then $\mu+\lambda$ with eigenvector $v$.
As for when to calculate one versus the other -- that's application-dependent. If you tell us what you need the eigenvector for, we might be able to give you advice.