So I have an example function:
$$f(a,b) = \frac 12 ((a+b)^2 + 3a + b)$$
It "encodes" two variables into one unique number.
Assuming a and b are always positive, how would you write a function to find two input variables of the "encoding" function? So basically, how would you write a function to "decode" the output of that "encoding" function?
Let $\mathbb N_0=\{0,1,2,\ldots\}$ and $\mathbb N=\{1,2,3,\ldots\}$.
To see this and how to reconstruct $(a,b)$ from $f(a,b)$, assume that $a$ and $b$ are nonnegative and call $s$ the largest integer $k$ such that $k^2\leqslant2f$, which can also be defined by the inequalities $$ s^2\leqslant2f\leqslant s(s+2). $$ Note that $2f=(a+b)^2+3a+b$ implies $(a+b)^2\leqslant2f\lt(a+b+2)^2$ hence $s=a+b$ or $s=a+b+1$.
Case $s=a+b$: this means that $2f\leqslant(a+b)(a+b+2)$, that is, $3a+b\leqslant2a+2b$, that is, $a\leqslant b$. Then, $a+b=s$ and $3a+b=2f-s^2$ hence $a=f-\frac12s(s+1)$ and $b=\frac12s(s+3)-f$. Then the condition $a\leqslant b$ holds and $a$ and $b$ are indeed integers.
Case $s=a+b+1$: this means that $2f\geqslant(a+b+1)^2$, that is, $3a+b\geqslant2a+2b+1$, that is, $a\geqslant b+1$. Then, $a+b+1=s$ and $3a+b=2f-(s-1)^2$ hence $a=f-\frac12s(s-1)$ and $b=\frac12s(s+1)-f-1$. Then the condition $a\geqslant b+1$ holds and $a$ and $b$ are indeed integers.
The first case applies when $s(s+1)\leqslant2f\leqslant s(s+2)$, the second when $s^2\leqslant 2f\lt s(s+1)$. To sum up:
Exemple: If $f=15123$ as in a comment, then $2f=30246$, $s=173$, $s^2+s=30102$, $2f\geqslant s^2+s$ hence we apply the second item above, which yields $a=15123-\frac12\cdot173\cdot174=72$ and $b=\frac12\cdot173\cdot176-15123=101$.