We have the function $f(x,y)=36xy$ and we want to maximize it subject to $3x+6y=m$. Using the Lagrange method I found that a critical point is $\left (\frac{m}{6}, \frac{m}{12}\right )$.
Using the second order condition and the Hessian matrix I found that that point is a saddle point.
For example the Hessian matrix is: \begin{equation*}H_F\left (\frac{m}{6}, \frac{m}{12}\right )=\begin{pmatrix}0 & 36 \\ 36 & 0\end{pmatrix}\end{equation*}
Then the eigenvalues are $\pm 36$.
Since we have positive and negative eigenvalues, the matrix is indefinite, and this means that the point is a saddle point.
Right?
But according to Wolfram it is a maximum point. (for example for m=40 ).
So, why do I get a saddle point?
Addressing intuition first
Here is a plot of your function, $f(x, y) = 36 * xy$. Note that there is only one critical point, and it is a saddle point.
The function
However, your restrict your domain to a line: $3x + 6y = m$, or: $y = \frac{1}{6} (m - 3x))$.
This makes your function $f(x, y) = 6 * (mx - 3x^2))$. A quadratic function! Note that varying $m$ is just sliding the constraint along the line's normal.
Let's plot $f(x, y)$ subject to that constraint for some arbitrarily chosen m.
Projected onto the constraint
By jove, a quadratic!
Both functions plotted
For $m=0$ the saddle point has become the maximum. For $m\ne0$, we just get a parallel line, indicating a parallel quadratic.
Addressing the Question
You want to compute
$\max_{x, y} 36 xy$
$s.t. A * \begin{bmatrix}x & y\end{bmatrix} = m$
for $A = \begin{bmatrix}3 \\ 6\end{bmatrix}$
The hessian of the function is the matrix you supplied. The "reduced hessian" is the hessian of the same beloved function along the degrees of freedom permitted by the constraints.
We can compute such a "reduced hessian" in one of two ways. The first is the matrix arithmetic approach, projecting onto the subspace where the constraint is satisfied. This is equivalent to projecting the original hessian into the nullspace of the constraint function.
A basis for the nullspace of A is trivial in 2D:
$Null(A) = Z = \begin{bmatrix}-1 \\ 1 / 2\end{bmatrix}$
And then $R = Z^T \begin{bmatrix} 0 & 36 \\ 36 & 0\end{bmatrix} Z = -36$.
$R$ is called the reduced hessian, and in our case is a scalar.
Another approach to constructing the same hessian is to project the function onto the constraint, and compute the hessian of that fella.
$f(x, y) = 6 * (mx - 3x^2))$
$\dfrac{df}{dx} f(x, y) = 6mx - 36x$ $\dfrac{df^2}{dx^2} f(x, y) = -36$
Which is consistent with our expectation.
This hessian is negative definite, indicating a maximum at the critical point.