There seems to be some disagreement (at least when consulting textbooks), what constitutes a symmetric operator and what constitutes a selfadjoint operator (of course, only the unbounded case is of relevance).
For example Lax, in his "Functional Analysis" book, says on pp. 354 that a symmetric operator $M:H\rightarrow H$ on a Hilbert space $H$ is one that fulfills $\left< Mx,y \right>=\left<x,My \right>$ for all $x,y\in H$ and on pp. 377 defines a selfadjoint operator to be an operator $M:D\subseteq H\rightarrow H$ with a dense subset $D$, such that $\left<Mx,y \right>=\left<x,My \right>$ for all $x,y\in D$. One the other hand, in Rudins book on pp. 331 the names for these properties are reversed, as they are also in Kreyszigs book (pp. 533)
Now this question isn't about what "ultimately" would mean for an operator to be symmetric resp. selfadjoint since there seems to be no agreement upon this issue.
But the majority of books (and some of the answers on math.SE concerning these definitions, as this and this one) seem to indicate, that the majority is on Rudin side.
My question therefore is, what prompted the "Lax"-side to switch the definitions ? Is there any advantage to this ?
EDIT Here are the quotes from these books:
Kreyszig (1978):
Definition Let $T:\mathscr{D}(T)\longrightarrow H$ be a linear operator which is densely defined in a complex Hilbert space $H$. Then $T$ is calles a symmetric linear operator if for all $x,y\in \mathscr{D}(T)$, $$\left< Tx,y\right>=\left<x,Ty\right>$$
$ $
Definition Let $T:\mathscr{D}(T)\longrightarrow H$ be a linear operator which is densely defined in a complex Hilbert space $H$. Then $T$ is calles a self-adjoint linear operator if $$T=T^*.$$
Lax (2002):
In this chapter we study operators $\boldsymbol{\operatorname{M}}$ that map a complex Hilbert space $H$ into itself, that are bounded, and that are symmetric in the sense that $\boldsymbol{\operatorname{M}}^*=\boldsymbol{\operatorname{M}}$. According to the definition of adjoint this means that for all $x$ and $y$ in $H$ $$\left<\boldsymbol{\operatorname{M}}x,y\right>=\left<x,\boldsymbol{\operatorname{M}}y\right>.$$
$ $
Definition Let $H$ be a complex Hilbert space. $D$ a dense subspace of $H$, and $A$ a linear operator defined on $D$. The adjoint $A^*$ of $A$ is the operator whose domain $D^*$ consists of all vectors $v$ in $H$ for which there is a vector denoted as $A^*v$ in $H$ such that $$\left<Au,v\right>=\left<u,A^*v\right>$$holds for all $u$ in $D$. Since $D$ is dense, for any given $v$ there can only be one such vector $A^*v$. Clearly, $D^*$ is a linear subspace of $H$, and $A^*$is a linear operator on $D^*$. $A$ is called self-adjoint if $D^*=D$ and $A^*=A$.
Ok, let me try to sort this out.
There is no disagreement between the definitions you quote. However, your initial paraphrases of those definitions are not accurate.
You wrote:
I have added the marks [A], [B] for reference.
I say that your [A] is what everyone calls "symmetric and bounded", your [B] is what everyone calls "symmetric", and that neither of them is what anyone calls "self-adjoint".
Your definition [A] does match Lax's first definition; however, in that definition he is defining what it means for a bounded operator to be symmetric. For a bounded operator defined on all of $H$, [A] and [B] are of course equivalent.
Your definition [B] does not match Lax's second definition; it is completely different. The key point is the introduction of the new subspace $D^*$ which in general may be different from $D$; Lax's requirement that $D = D^*$ is crucial and by no means trivial. Lax's second definition agrees with every other definition of "self-adjoint" (for unbounded operators) that I have ever seen.
Kreyszig's first definition matches your definition [B], which as I claimed is what people call "symmetric".
Kreyszig's second definition matches neither your [A] nor [B], but if you investigate his definition of $T^*$, I claim you will find it equivalent to Lax's second definition. It is again the standard definition of "self-adjoint".
Edit. I'll address your added comments.
Let me first mention that when working with unbounded operators, one has to keep in mind that an unbounded operator is really a pair $(A,D)$ of an operator and its domain: a linear subspace $D \subset H$ and a linear map $A: D \to H$. In particular, for two operators to be equal, they have to have the same domain. The fact that people often just use the word "operator" and talk mostly about $A$ instead of $D$ tends to obscure this essential point. For the rest of this answer, I'll write the operator and its domain explicitly.
So suppose $(A,D)$ is an unbounded operator and $D$ is dense (people say "$A$ is densely defined"). As you say, we define a new operator $(A^*, D^*)$, called the adjoint of $(A,D)$, as in Lax: $v \in D^*$ iff there is a vector $w_v \in H$ such that for every $u \in D$ we have $\langle Au, v \rangle = \langle u, w_v \rangle$. For each $v \in D^*$, if $w_v$ exists then it is unique, and we define $A^*v := w_v$. We say $(A,D)$ is self-adjoint if $(A,D) = (A^*, D^*)$, i.e. $D=D^*$ and $A=A^*$.
It is true that we have the relationship $$\langle Au, v \rangle = \langle u, A^* v \rangle, \quad \forall u \in D, v \in D^* \quad \quad (\star)$$ But I want to emphasize that $(A^*, D^*)$ is not just some operator satisfying $(\star)$; it is the specific operator defined in the previous paragraph. (In some sense, it is the operator satisfying $(\star)$ and having largest possible domain.)
Then, as you have argued, if $(A,D)$ is self-adjoint, then in particular [B] holds: we have $\langle Au, v \rangle = \langle u, A v \rangle$ for every $u,v \in D$. That is to say, if $A$ is self-adjoint then it is symmetric.
The converse is false: a symmetric operator need not be self-adjoint. Here is a typical counterexample.
Let $H = L^2([0,1])$, and let $D = C^2_c((0,1))$. That is, $D$ consists of $C^2$ functions having compact support contained in $(0,1)$. In particular, if $f \in D$ then $f$ and all its derivatives vanish at $0$ and $1$. For $f \in D$, let $Af = f''$ be the second derivative of $f$.
Recall from calculus that for any $f,g \in C^2([0,1])$ we have, by integrating by parts twice: $$\int_0^1 f''(x) g(x)\,dx = f'(1) g(1) - f'(0) g(0) - f(1)g'(1) + f(0)g'(0) + \int_0^1 f(x) g''(x)\,dx.$$ If $f,g \in D$, then all the boundary terms vanish and we have $\int_0^1 f''(x) g(x)\,dx = \int_0^1 f(x) g''(x)\,dx$. That is, for all $f,g \in D$, $\langle Af, g \rangle = \langle f, Ag \rangle$. So $(A,D)$ is symmetric.
But I claim that $(A,D)$ is not self-adjoint; in particular, $D^*$ is a proper superset of $D$. Indeed, we can see that if $f \in D$ and $g \in C^2([0,1])$, then again all boundary terms vanish and $\int_0^1 f''(x) g(x)\,dx = \int_0^1 f(x) g''(x)\,dx$, i.e. $\langle Af, g \rangle = \langle f, g'' \rangle$. So $g \in D^*$ and $A^* g = g''$. Thus $C^2([0,1]) \subset D^*$: we see that $D^*$ contains functions which do not vanish at 0 and 1 and hence are not in $D$. (In fact, $D^*$ is the Sobolev space $H^2([0,1])$, though it takes a little more work to show this.) So $(A,D) \ne (A^*, D^*)$.
Moreover, $(A^*, D^*)$ is not symmetric. Take for example $f(x) = x$ and $g(x) = x^2$. Since $f,g$ are $C^2$ they are contained in $D^*$ and $A^*f = f'' = 0$, $A^*g = g'' = 2$. Then you can check that $\langle A^* f, g \rangle = 0$ while $\langle f, A^*g \rangle = 1$.
(Actually, you can show that if $(A,D)$ and $(A^*, D^*)$ are both symmetric, then $(A^*, D^*)$ is self-adjoint, and in fact $(A^*, D^*)$ equals the closure of $(A,D)$. In this case $(A,D)$ is said to be essentially self-adjoint.)