The matrix $\mathbf{A}$, written in the form of column vectors, is
$\mathbf{A}=(\mathbf{a}_1, \mathbf{a}_2,...\mathbf{a}_P)$
where $\mathbf{a}_k$ is an $M\times 1$ complex column vector that is given by
$\mathbf{a}_k = (1,e^{j\tau_k},e^{j2\tau_k},\cdots,e^{j(M-1)\tau_k})$,
$j＝\sqrt{-1}$ and all real constants $\tau_k$, $k=1,2,\cdots,P$, are unequal.
Assume $M>P$, then what properties, such as positive definiteness, eigenvalue decomposition, matrix rank etc, do the matrix $\mathbf{A}^H\mathbf{A}$ have? And how to prove these properties?
The matrix $A^HA$ is positive-semidefinite, which is true even without extra assumptions on $A$. It's Hermitian, since $$(A^HA)^H = A^H(A^H)^H = A^H A.$$ It's positive-semidefinite because, for any vector $v \in \Bbb{C}^P$, $$v^H(A^HA)v = (Av)^H(Av) = \|Av\|_2^2 \ge 0.$$ In order for it to be positive-definite, whenever $v \neq 0$, we need $\|Av\|_2 > 0$. This happens if and only if the columns of $A$ are linearly independent.
When are the columns linearly independent? Note that a column of $A$ takes the form: $$(1, \omega_k, \omega_k^2, \ldots, \omega_k^{M-1}),$$ where $\omega_k = e^{j\tau_k}$. This makes $A$ the transpose of a Vandermonde matrix, whose rows are linearly independent provided they are distinct, and do not outnumber the columns. So, your matrix will be positive-definite if and only if the $\omega_k$ numbers are pairwise distinct (proving this is usually done by extending the Vandermonde matrix to be square, and computing the determinant, which elegantly establishes the above claim).
Of course, you assume the $\tau_k$ numbers are pairwise distinct, but this is not quite sufficient, as the complex exponential is not injective. In fact, the complex exponential has a period of $2\pi$. So, your matrix will be positive-definite if and only if $\tau_k - \tau_l$ is only an integer multiple of $2\pi$ if $k = l$.
As for rank, the rank of $A$ and the rank of $A^HA$ agree (once again, without further assumption on $A$). This can most easily be seen by the rank-nullity theorem. The nullspace of $A^HA$ actually matches the nullspace of $A$, since $$Av = 0 \implies A^HAv = 0 \implies v^HA^HAv = 0 \implies \|Av\|^2_2 = 0 \implies Av = 0,$$ so $Av = 0 \iff A^HAv = 0$. That is, the nullspaces agree. Since both matrices have $P$ columns, the ranks also agree.
The rank of $A$ will just be, once again, the number of distinct columns, due to the Vandermonde result. Thus, the rank of $A^H A$ will be the number of distinct $\omega_k$s. Equivalently, it will be the number of equivalence classes of the equivalence relation $x \sim y \iff (x - y)/(2\pi) \in \Bbb{Z}$ when applied to $\{\tau_1, \ldots, \tau_P\}$.
As for eigenvalues/eigenvectors, I greatly doubt that there will be a "nice" formulation for eigenvalues and eigenvectors. Obviously, if columns repeat, then we will get some $0$ eigenvalues. If $\omega_k = \omega_l$ for some $k \neq l$, then to get an eigenvector, we simply place a $1$ in the $k$th position, and $-1$ in the $l$th position, and $0$s elsewhere. For non-zero eigenvalues/eigenvectors, I don't think we'll get anything neat.