I'm trying to maximize the following expression $f$ algebraically. I'm not entirely sure if it can be done without approximating the solution by removing the floor functions so I'm asking here.
$$f(x,y) = \left(C - (x+y)\right)\sqrt{\lfloor (A+x)(1+0.1H)\rfloor\lfloor (B+y)(1.1-0.1H)\rfloor}$$
where constants $A,B,C\in\mathbb{Z}^+$, variables $x, y\in\mathbb{Z}^+$ with $x+y<C$ and $H$ is chosen to be either $0$ or $1$, whichever makes $f$ larger ($H$ still depends on $x$ and $y$ so it doesn't need to be a 3rd variable).
Although $x$ and $y$ are integers, I thought that perhaps an approach with calculus by treating them as reals would be viable if the expression was rearranged.
What I've tried:
Let $u = A + x$, $v = B + y$ and $k = A + B + C$.
Then
$$f = (k - (u+v))\sqrt{\lfloor u(1+0.1H)\rfloor\lfloor v(1.1-0.1H)\rfloor}$$
$$=(k - (u+v))\sqrt{(u+\lfloor0.1u\rfloor H)(v+\lfloor0.1v\rfloor(1-H))}$$
$$=(k - (u+v))\sqrt{uv+u\lfloor0.1v\rfloor(1-H)+v\lfloor0.1u\rfloor H}$$
We now see $H$ is determined by which of the terms $u\lfloor0.1v\rfloor$ and $v\lfloor0.1u\rfloor$ is larger. The smaller term will vanish, leaving the larger term in the expression.
Hence we can conclude
$$H = \theta(v\lfloor0.1u\rfloor-u\lfloor0.1v\rfloor)$$
where $\theta$ is the Heaviside function with $\theta(0) = 1$
Let $u=10p+q$ and $v=10r+s$ where $p,q,r,s\in\mathbb{Z}\,,\,\,0\leqslant q, s<10$.
Then
$$H = \theta(\lfloor p+0.1q\rfloor v-\lfloor r + 0.1s\rfloor u)$$
$$= \theta(pv - ru + \lfloor0.1q\rfloor v-\lfloor0.1s\rfloor u)$$
$$= \theta(p(10r+s) - r(10p+q))$$
$$= \theta(ps - rq)$$
$$= \theta\begin{vmatrix}p & q \\ r & s\end{vmatrix}$$
And now I'm stuck. I'm not sure if what I've just shown is helpful, but perhaps the next step is to turn it into a 4 variable problem with $p,q,r,s$?
If it is not possible to do it analytically, then I would still be interested in a computational method which still does as much of the question analytically as possible.
This is a partial answer, but I hope it helps.
Using your change of variables, let $$f(u,v)=(k-(u+v))\sqrt{uv+u\lfloor{0.1v}\rfloor(1-H)+v\lfloor{0.1u}\rfloor H}.$$ Now, using that $u\lfloor{0.1v}\rfloor(1-H)\leq0.1uv$ and $v\lfloor{0.1u}H\leq0.1uv$, we have that $$f(u,v)\leq g(u,v):= (k-(u+v))\sqrt{1.1uv}.$$ We'll assume for now that $u,v\in\mathbb{R}$.
If we fix $t=x+y$, then the maximum of $xy$ is attained when $x=y=\frac{t}{2}$, so that we can look for the maximum of the function $h(t):=\frac{\sqrt{1.1}}{2}(k-t)t$. This maximum is attained at $t=\frac{k}{2}$, so that the maximum of $g$ is attained when $u=v=\frac{k}{4}$.
So, an heuristic method to solve your problem is to look for integer pairs $(x,y)$ near from $(\frac{k}{4}-A,\frac{k}{4}-B)$. When $A,B\geq\frac{k}{4}$, the maximum is attained at $(1,1)$, which is the closest you can get to $(\frac{k}{4}-A,\frac{k}{4}-B)$. When $A\leq\frac{k}{4}\leq B$, then, the minimum will be of the form $(1,y)$, and analogously for the inverse inequality.