Let's say I have a quadratic problem with nonnegative triangular matrix Q and binary decision variables x.
$$min_{x} f(x) = x^\top Q x$$ $$x \in [0,1]$$ $$\forall i,j \ q_{ij}\geq 0$$
AFAIK, Only the relaxed problem with $0 \leq x \leq 1$ can be convex, because binary variables are not a convex set.
Because the matrix Q is not positive semidefinite, the objective function is not convex per se. But because the problem domain is positive ($x \geq 0$) and the matrix is nonnegative ($q_{ij} \geq 0$), the range is nonnegative ($f(x)\geq 0$) which means the problem is convex in the domain. Does that mean the problem is quasiconvex or is this defined as a convex problem?
I'm a bit confused about the necessity of domain-restrictions for quasiconvexity. Or to reformulate: Is an optimization problem which is convex for all input variables convex?
I have understood that it is possible to make all quadratic cost functions convex by adding a large enough number to Q so that f(x) is always positive (see Binary quadratic optimization problem). But if my problem is already (quasi-)convex, this would not be necessary.
For a problem "$\min f(x)$ subject to $x \in S$" to be convex, the domain $S$ needs to be a convex set and the objective function $f(x)$ needs to be convex over the domain. For the problem to be considered quasi-convex, the domain $S$ needs to be a convex set, and the objective function $f(x)$ needs to be quasi-convex.
For your particular program, the domain $[0,1]^n$ is convex. However, from the conditions that $Q$ is triangular and has non-negative entries, it isn't guaranteed that the objective function $f(x) = x^TQx$ is convex or even quasi-convex.
For example, if $Q = \begin{bmatrix}1&10\\0&1\end{bmatrix}$, then $f\left(\begin{bmatrix}1\\0\end{bmatrix}\right) = f\left(\begin{bmatrix}0\\1\end{bmatrix}\right) = 1$ but $f\left(\begin{bmatrix}1/2\\1/2\end{bmatrix}\right) = 3$.
So the set $\{x \in [0,1]^2 : f(x) \le 2\}$ contains the points $\begin{bmatrix}1\\0\end{bmatrix}$ and $\begin{bmatrix}0\\1\end{bmatrix}$ but not $\begin{bmatrix}1/2\\1/2\end{bmatrix}$.
Thus, $\{x \in [0,1]^2 : f(x) \le 2\}$ is not convex. Hence, $f(x)$ is not quasi-convex.
Also, your problem as currently written is trivial. You've already shown that $f(x) = x^TQx \ge 0$ since the decision variables $x_i$ and matrix elements $q_{ij}$ are nonnegative. Then, since $f(\vec{0}) = 0$, where $\vec{0} \in [0,1]^n$, we have that the minimum is in fact $0$, which is obtained when $x = \vec{0}$.