(Notation $R[[x]]$ means the ring of formal power series over $R$, a commutative ring with unity.)
Observations:
Since $A\to B$ integral implies $A[x]\to B[x]$ integral, $B[x]\subset B[[x]]$ is integral over $A[[x]]$. Thus it would suffice to show that the integral closure of $A[[x]]$ in $B[[x]]$ is closed in the $(x)$-adic topology.
Since we have a commutative diagram $$ \require{AMScd} \begin{CD} A[x]@>>> B[x]\\ @VVV@VVV\\ A[[x]]@>>> B[[x]] \end{CD} $$ it would suffice to show that $B[x]\to B[[x]]$ is integral, for $A[x]\to B[[x]]$ would be integral too and then we may apply [02JM]. However, I don't have hope this is true (update: red_trumpet explained why this is false in a comment).
If $A\to B$ is finite, then $A[[x]]\to B[[x]]$ is finite too (and in particular integral). Thus, a counterexample must come from a non-finite integral morphism $A\to B$ (equivalently, integral but not of finite type). Indeed, if $A\to B$ is finite, then so is $A[x]\to B[x]$. Thus there is a surjection $A[x]^{\oplus n}\to B[x]$ of $A[x]$-modules. If $(x)\subset A[x]$, then $(x)$-adic completion of $A[x]$-modules gives a surjection $A[[x]]^{\oplus n}\to B[[x]]$ of $A[[x]]$-modules [0315], i.e., $A[[x]]\to B[[x]]$ is finite (analogously, one in general proves that finite modules are stable under adic completions).
I suspect not every element of $B[[x]]$ will be integral over $A[[x]]$. Consider $A=K$ a field and $B=\overline{K}$ an algebraic closure. Suppose $\overline{K}/K$ is not finite (for instance, this happens if $K$ is a finite field). Then, for non-zero $a\in K$, my guess is that the formal power series $\sum_{i\geq 1} a^{1/i}x^i\in\overline{K}[[x]]$ won't be integral over $K[[x]]$. However, I am unable to prove it. I tried by induction on the degree of the integral relation equation and it doesn't work.