You are given the following expression:
$$(ax + by)^{n} = -15120x^{4}y^{3}$$
Determine the constants $a$, $b$ and $n$.
My attempt to solve this problem is by trying to use the binomial theorem backwards.
The binomial theorem: $(ax + by)^{n} = \displaystyle\sum_{k = 0}^{n} {\binom{n}{k}(ax)^{n-k}(by)^{k}}$
And so if you compare the expressions, you get:
$$\binom{n}{k}(a)^{n-k}(b)^{k} = -15120$$
$n - k = 4$ and $k = 3$ so that $n = 7$
Here is where I get stuck, because now I have:
$$\binom{7}{3}(a)^{4}(b)^{3} = -15120$$
Two unknowns... How to solve it? Am I even doing it correctly?
The problem with your approach is that you have equated the coefficients of the $x^4y^3$ term on both sides, as if this is an equation of polynomials, but you have ignored the remaining terms.
But this cannot be an equation of polynomials to begin with, because most terms on the RHS are missing which would need to be there. For example $a^nx^n$ and $b^ny^n$.
So are we to assume $x$ and $y$ are some fixed constants?