On page 16 of Prof. Jones' online von Neumann algebras notes(you can find the book here: von Neumann algebras notes
There is an exercise 3.3.9(iv) (labeled as a harder one),
Suppose $H$ is a separable Hilbert space, show that there is no nonzero linear map $tr: B(H)\rightarrow \mathbb{C}$ satisfying $~tr(ab)=tr(ba)$.
I can only figure out that if such $tr$ exists, it must be zero on any finite rank operator by using the right shift operator $S$, but how to process from finite dimensional operators to all operators in $B(H)$ by just using the property $tr(ab)=tr(ba)$, can any one give some hint?
The same idea that you use to show that $tr$ is zero on finite-rank operators, can be used to see that it is zero on any projection.
Consider a projection $q$ with infinite dimension and codimension. Consider an orthonormal basis $\{\xi_n\}$ such that $\{\xi_{2n}\}$ spans the range of $1-q$. Let $\{e_{kj}\}$ be the matrix units associated with our orthonormal basis, and let $$ x=\sum_je_{j,2j}. $$ Then $$ x^*x=\sum_je_{2j,2j}=1-q,\ \ xx^*=\sum_je_{j,j}=1. $$ So $tr(1)=tr(xx^*)=tr(x^*x)=tr(1-q)$. Then $tr(q)=0$. As $q$ was any projection with infinite dimension and co-dimension, we also have $tr(1-q)=0$. Thus $$ tr(1)=tr(q+(1-q))=tr(q)+tr(1-q)=0+0=0. $$ Now if $p$ is a projection with finite co-dimension, then $tr(1-p)=0$ (as $1-p$ is finite-rank), but then $tr(p)=0$ since $tr(1)=0$.
We have shown that $tr(p)=0$ for all projections in $B(H)$. It is well-known (see here for references) that every operator in $B(H)$ is a finite linear combination of projections. So $tr=0$.
(in part iii one can avoid this last argument: the fact that $tr$ is positive allows one to use Cauchy-Schwarz, and this together with $tr(1)=0$ can be used to see that $tr=0$: indeed, in that case for any $x$ $$ |tr(x)|=|tr(1x)|\leq tr(1)^{1/2}tr(x^*x)^{1/2}=0. $$ so $tr=0$)