9.16. a. Given a symmetric Hermitian quadratic form $A(x, x)$ in an $n$-dimensional complex space $\mathbf C_n$, there exists a basis in $\mathbf C_n$ in which $A(x, x)$ can be written in the canonical form $$A(x, x) = \sum_{k = 1}^n \lambda_k\,\eta_k\,\bar\eta_k = \sum_{k = 1}^n \lambda_k\,|\eta_k|^2$$ with real coefficients $\lambda_1, \lambda_2, \cdots, \lambda_n$.
In the proof of this proposition, the book says, on page 251, that the following transformation $$\xi_1 = \xi_1' + \xi_2',\\ \xi_2 = \xi_1' + i\,\xi_2'$$ carries the sum $a_{12}\xi_1 \bar{\xi_2} + \bar a_{12} \bar{\xi_1} \xi_2 \;(a_{12} \neq 0)$ into the expression $$(a_{12} + \bar a_{12}) \xi_1' \bar{\xi_1'} - i(a_{12} - \bar a_{12}) \xi_2' \bar{\xi_2'} + \cdots.$$ Note the ellipsis at the end of the above expression. The complete expression is $$(a_{12} + \bar a_{12}) \xi_1' \bar{\xi_1'} - i(a_{12} - \bar a_{12}) \xi_2' \bar{\xi_2'} + (a_{12} + i\bar a_{12}) \bar{\xi_1'} \xi_2' + (\bar a_{12} - i a_{12}) \xi_1' \bar{\xi_2'}.$$
But, in my understanding, we should find a transformation that could carry the sum into a canonical form like $\lambda_1 \xi'_1 \bar{\xi_1'} +\lambda_2 \xi_2' \bar{\xi_2'}$, where $\lambda_1, \lambda_2$ are real.
And, if I'm right, how can we construct such a transformation? Thank you!
The construction is already done on p.250, the page preceding p.251. Given $A(x,x)$, there are only three possibilities:
Shilov has dealt with case 1 on p.250, and nothing needs to be done in case 2 because $A(x,x)=0$ in that case.
The purpose of the transformation on p.251 isn't to express $A(x,x)$ as a sum of squares, but to reduce case 3 to case 1. By relabelling the indices if necessary, we may assume that $a_{12}\ne0$. It follows that at least one of $a_{12}+\bar{a}_{12}$ or $i(a_{12}+\bar{a}_{12})$ is nonzero. Therefore, by the change of variables mentioned in your question, we may write $$ A(x,x)=\underbrace{(a_{12}+\bar{a}_{12})}_{a_{11}'}\,\xi_1'\bar\xi_1' \, \underbrace{- i(a_{12}-\bar{a}_{12})}_{a_{22}'}\,\xi_2'\bar\xi_2' + \cdots $$ and now at least one of $a_{11}'$ or $a_{22}'$ is nonzero among the new coefficients $a_{ij}'$.