I have a function $f$ defined as follows,
$$f(n) = \sum_{1 \leq i \leq n} i(i-1)$$
but I want to find $n$ and $n'$ for which the following holds,
$$\frac{f(n)}{n(n-1)} = \frac{3f(n'/3)+6f(2n'/3)}{n'(n'-1)}$$
I tried a few random numbers, but I failed to solve this manually. I was wondering if there is a systematic way of solving this or is there a solution to this at all? (For simplicity, feel free to assume that $n$ and $n'$ are both divisible by 2 and 3).
Let's go ahead and compute the sum. $$f(n) = \sum_{i=1}^n (i^2 - i) = \frac{1}{6}n(n+1)(2n+1) - \frac{1}{2}n(n+1) = \frac{1}{3}n(n+1)\left(n-1\right)$$
Then $\frac{f(n)}{n(n-1)} = \frac{n+1}{3}$. Now you can compute $f(2n'/3)$ and $f(n'/3)$ and reduce it to a simple equation in $n$.
Indeed, we get $3f(n'/3) = \frac{n'}{3}\left(\frac{n'}{3} + 1\right)\left(\frac{n'}{3} - 1\right) = \frac{n'}{27}\left(n'+3\right)(n' - 3)$ and $6f(2n'/3) = \frac{4n'}{27}\left(2n' + 3\right)(2n' - 3)$
So that $$f(n'/3) + f(2n'/3) = \frac{n'}{27}(17n'^2 - 45)$$ and hence your equation reduces down to $$9(n+1)(n'-1) = 17n'^2 - 45$$ which is a quadratic in $n'$.