The question is related to exercise 6.F in Olssons's "Algebraic Spaces And Stacks" and the 2 examples on page 14 of https://people.math.ethz.ch/~pink/ftp/FGS/CompleteNotes.pdf and in general to group actions on schemes.
Consider the action of $G=\mathbb G_{m,k}$ on $\mathbb A_k^n$ given by $t(x_1,\ldots,x_n)=(tx_1,\ldots,tx_n)$. Then $B=A^G=k$.
I can see this 'algebraically', by writing explicitly the map $k[x_1,\ldots,x_n]\to k[t,t^{-1}]\otimes_kk[x_1,\ldots,x_n]$. If I am not mistaken, the map sends $x_i\to t\otimes x_i$, thus $m(a)=1\otimes a$ implies $a\in k$.
If we 'restrict' to $U=\mathbb G_{m,k}\times\mathbb A_k^{n-1}$, then $m(a)=1\otimes a$ also for $a=x_i/x_1$, and Example 2 follows.
Could someone elaborate this for me in a more geometric way? (via 'bad' orbits/closure maybe) I hope this is not too vague, but for example the last 3 lines in the link in Example 1 don't make much sense to me, and similarly, I don't really understand (other than working through the algebra as above) why restricting to $U$ things 'suddenly work'. Feel free to explain in terms of more general actions.
Let me assume that $G$ is reductive (e.g a torus or a simple algebraic group) and that $k$ has characteristic zero (I'm not sure what happens in positive characteristic).
The geometric interpretation is now very simple : closed points in $A^G$ should corresponds should corresponds to closed orbits in $X = \text{Spec}(A)$.
For example, if $G$ is finite, then the closed points of $X/G$ are in 1-1correspondance with the orbits of the action. But this is a very rare case : typically, if $G = GL_n$ acts on $M_n$ by conjugaison, the fiber over $0$ is composed of the set of nilpotent matrices which certainly corresponds to several orbits. The closed orbit is $\{0\}$.
Here the orbits are punctured lines which are not closed, the unique closed orbit is the origin. This is why $A^G = k$. Notice that if you delete the origin, you now get something nicer, the projective space, because punctured lines are closed in $\Bbb A^{n+1}_k \backslash \{(0,\dots, 0)\}$, so in this case the quotient indeed parametrize lines passing through the origin.