I searched for some results online but it seems that the objective functions considered there depend on continuous variables. I am not sure if it will be different when the variables are restricted to be integers.
So, for example, we have the following objective function $$\frac{\sum_{i=1}^n (x_iy_i)}{\sqrt{\sum_{i=1}^n x_i} \sqrt{\sum_{i=1}^n y_i}},$$ where $x_i, y_i \in \{0,1\}$.
Clearly, this function is non-linear. How can we determine if it is convex or not? I appreciate it if you can provide some general guidance.
Also, I guess this optimization problem falls into (non-linear) Binary-Integer-Programming. Can you provide some references on how to start solving such a class of problems?