I have this known matrix B with 9 columns and 30 rows. I am then suppose to minimize the following problem with respect to h in MATLAB, but I really don't know how to do it:
$min ||B*h||_2^2$, where $||h| =1$
I have only ever tried doing this with Ax-b where the solution then would be found as : $x = (A'A)^{-1}A'b$
But I am thinking that it might have something do to with SVD.
The above problem is Boolean quadratic programming(a kind of quadratic constrained quadratic programming as long as matrix B is positive semi-definite). You can try convex optimization package for MATLAB (from http://cvxr.com/cvx/) .
You may also check this http://cvxr.com/cvx/examples/cvxbook/Ch05_duality/html/qcqp.html