I saw the following notation in a proof that $\lim_{n \to \infty} (\cos{\frac{x}{\sqrt{n}}})^{n}=e^{-x^{2}/2}$ but I don't understand what it means.
$\log{(\cos{\frac{x}{\sqrt{n}}})}=\log{(1-\frac{x^{2}}{2n}}+O(\frac{x^{4}}{n^{2}}))=-\frac{x^{2}}{2n}+O(\frac{x^{4}}{n^2})$
Could someone explain what this means?
It's a notation for the remainder term.
For example:
for $x$ ~ $0$ : $e^x = 1 + x + \dfrac{x^2}{2} + O(x^3)$, because the remainder has a growth rate similar to $x^3$ (in this case, the remainder is $\dfrac{x^3}{3!} + \dfrac{x^4}{4!} + ...)$
The formal definition, for two functions $f$ and $g$ about a point $a$, is that:
$f = O(g)$ if $\exists k > 0$ such that, for $x$ in the neighbourhood of $a$, $f(x) \leq k*g(x)$.
Edit (thanks to mathworker21) : hence, with $h$ a function, the notation $f = h + O(g)$ means that $f$ equals $h$ plus some remainder term which has a growth rate similar to that of $g$. Hence why it is so often used in Taylor expansions: we often approximate the function up to some term, and need a way to rigorously discard the higher terms.
You'll probably often see the alternative notation $f = o(g)$, which means $f$ is neglictible against $g$ (ie, $g$ grows much faster), which is defined, about a point $a$, as :
$\lim\limits_{x\to a} \dfrac{f(x)}{g(x)} = 0$.
You can then rewrite the Taylor expansion as $e^x = 1 + x + \dfrac{x^2}{2} + o(x^2)$.