Given a function $f = ax - by + 0c$, with constraints $y\geq 0$, $x\geq 0$, $c\geq 0$, and $x+y+c=n$, where all are integers. I am interested to find the count of all unique values of $f$.
I know that total number of different arrangements of $a$, $b$, $c$ are $(n+1)(n+2)/2$. But how do I know how many these arrangements have same value of $f$?
If $a$ and $b$ have a common factor then so does every $f$. So divide $a$ and $b$ by the common factor. From now on, $a$ and $b$ are coprime.
$(x,y)$ lies within a triangle. Note $f=ax-by+0c=a(x-b)-b(y-a)+0c$ so only count $(x,y)$ if $(x-b,y-a)$ lies outside the triangle, otherwise the same $f$ is counted twice.
Suppose $n=5$, $a=2$ and $b=3$. The triangle has vertices $(0,0), (0,5)$ and $(5,0)$. The bottom three rows should all be counted because $x-3\lt0$ which is outside the triangle. For the other rows, only the first two columns should be counted.