An algorithm solves a problem of size $n$ by recursively calling
8 subproblems, with each subproblem of 1/4 the size of the original input.
It then combines their solutions to form the solution of the original problem in $\mathbf{O(n\cdot sqrt(n))}$ time.
I already have the recurrence equation of $8 \cdot T(\frac{n}{4})+O(n\cdot\sqrt{n})$ figured out but I have no idea how to calculate the Big-O portion. I know it has something to do with the master theorem/iteration method, but how would I go about doing this?
As you stated we can solve the recurrence using the iteration method. Given the recurrence:
$$ T(n) = 8 T \left( \frac{n}{4} \right) + n^{\frac{3}{2}} $$
We expand the first few terms getting:
$$ T(n) = n^{\frac{3}{2}} + 8 \left( \frac{n}{4} \right)^{\frac{3}{2}} + 8^2 \left( \frac{n}{4^2} \right)^{\frac{3}{2}} + 8^3 \left( \frac{n}{4^3} \right)^{\frac{3}{2}} + ... + 8^{\log_4 n} T(1) $$
Rewriting we get:
$$ T(n) = \sum_{k = 0}^{\log_4 n} \left( \frac{8}{4^{\frac{3}{2}}} \right)^k n^{\frac{3}{2}} = n^{\frac{3}{2}} \sum_{k = 0}^{\log_4 n} 1 = n^{\frac{3}{2}} \log_4 n $$
And so we conclude:
$$ T(n) = O\left(n \sqrt{n} \cdot \log_4 n\right) $$
We could have also used the master theorem to deduce the solution. The master theorem applies to recurrences of the form:
$$ T(n) = a T \left( \frac{n}{b} \right) + f(n) $$
In our problem $a = 8$, $b = 4$ and $f(n) = n \sqrt{n}$. This means $\log_b a = \log_4 8 = \frac{3}{2}$ and $f(n) = n^{\log_b a}$ and so the second case of the master theorem applies. Then, the master theorem states the solution to the recurrence to be:
$$ T(n) = O\left( n \sqrt{n} \cdot \log_2 n \right) $$