Let $f:O\subset \Bbb{R}^n\to\Bbb{R}$ be a $C^3$ function. Let $x_0\in O$ be a critical point of $f$. Let \begin{align}A=\left(\frac{\partial ^2f}{\partial x_i\partial x_j}\right)_{1\leq i,j\leq n}\end{align} If $A$ is positive, then $x_0$ is a local minimum.
Proof
By Taylor's formula \begin{align}f(x)=f(x_0)+f'(x_0)(x-x_0)+\frac{f''(x_0)}{2!}(x-x_0)^2+\Vert x-x_0 \Vert^2\epsilon(x-x_0)\end{align} But \begin{align}f'(x_0)=0\end{align}This implies \begin{align}f(x)-f(x_0)=\frac{f''(x_0)}{2!}(x-x_0)^2+\Vert x-x_0 \Vert^2\epsilon(x-x_0)\end{align} Now, \begin{align} f''(x_0)(x-x_0)^2=\langle A(x-x_0),(x-x_0)\rangle\end{align} \begin{align} =\sum^{n}_{i=1}\sum^{n}_{j=1}\frac{\partial ^2f(x_0)}{\partial x_i\partial x_j}(x_j-x_{0_j})^2\end{align} \begin{align} =\sum^{n}_{i=1}\sum^{n}_{j=1}(x_j-x_{0_j})\frac{\partial ^2f(x_0)}{\partial x_i\partial x_j}(x_j-x_{0_j})\end{align} \begin{align} =\langle (x-x_0),A(x-x_0)\rangle\end{align} Thus, \begin{align}f(x)-f(x_0)=\frac{\langle (x-x_0),A(x-x_0)\rangle}{2!}(x-x_0)^2+\Vert x-x_0 \Vert^2\epsilon(x-x_0)\end{align} \begin{align}f(x)-f(x_0)\geq\Vert x-x_0\Vert^2\left[\alpha+\epsilon(x-x_0)\right],\;\;\text{for some}\;\alpha>0\end{align} Hence, $\exists\;\delta>0$ such that $\Vert x-x_0\Vert<\delta$ implies $\left[\alpha+\epsilon(x-x_0)\right]>0$. So, for $x\in(x_0-\delta,x_0+\delta)$ \begin{align}f(x)\geq f(x_0)\end{align} Therefore, $x_0$ is a local minimum.
Can someone check if this proof is correct? Corrections will be highly welcome! Thanks
A few corrections/elaborations for you: First of all, you want $O$ to be an open subset of $\Bbb R^n$ (or at least to contain a ball around $x_0$). Second, by $A$ you mean the Hessian matrix evaluated at $x_0$. Third, what do you mean by saying $A$ is positive? Usually we say positive definite. Fourth, your notation for the second-degree term in Taylor's formula really doesn't make sense. (Is your textbook using that?) It really needs to be what you wrote down after the "Now." :) Fifth, the $\|$ at the end of the displayed equation doesn't belong there, but you do want to say that $\epsilon(u)\to 0$ as $u\to 0$.
Now, I'm not sure what the point of your "Now" computation is. But you do need to explain that if the matrix $A$ is positive definite, then $\langle Av,v\rangle \ge \alpha\|v\|^2$ for some positive $\alpha$. (This follows in one of two ways: (1) all the eigenvalues of $A$ are positive; or (2) the continuous function $Q(v)=\langle Av,v\rangle$ on the unit sphere has a minimum value, which must of course be positive. And there's your $\alpha$.)
Last, I would suggest being more explicit in your last step. By the limit definition with $\epsilon$, noting that $\alpha>0$, you know that there's $\delta>0$ so that whenever $\|x-x_0\|<\delta$ we'll have $|\epsilon(x-x_0)|<\alpha$. That'll do it. (Why?)
Overall, with a bit of revision, good job! :)