Here is what I have tried, but I am having trouble completing the last step, i.e.
How do I find the $$\sum_{i=0}^{log_\frac{4}{3}(n-1)}{\left(\frac{5}{4}\right)^{2id}}$$
If I treat it as constant, the solution does not generalize, I guess. Your help is appreciated.
edit_1: couple of mistakes in the image
there should be $T(1).(4/3)^{log_{4/3}n} $ at the work at the last layer=leaves layer
there should be $ + O(n)$ after each of the step at the bottom left corner
edit_2: possible answer
If you look through CLRS, there is an exercise just after the proof of the master theorem 4.6.2
And if we apply the answer of 4.6.2 here, then answer to this problem would be $n^d*log_{4/3}n$. It makes sense to argue on these lines. The work done at each level times the number of levels.
But still how to prove it, so that it is formal?
You want to solve the recurrence $$T(x)=T\left(\frac{x}2\right)+T\left(\frac{3x}{4}\right)+x^d$$
We use the Akra-Bazzi method to determine the asymptotic behavior of $T(x)$. We have:
$$g(x)=x^d\\k=2\\a_1=a_2=1\\b_1=\frac12\quad b_2=\frac34\\h_1(x)=h_2(x)=0$$
We find the value $p$ for which $a_1b_1^p+a_2b_2^p=1$. It turns out that $p\approx 1.507$
The Akra-Bazzi method then states that $$T(x)\in \Theta\left(x^p\left(1+\int_1^x\frac{g(u)}{u^{p+1}}\,du\right)\right)$$
Evaluating the integral, we have that $$T(x)\in\Theta\left(x^p\left(1+\int_1^xu^{d-p-1}\,du \right)\right)\\ T(x)\in\Theta\left(x^p\left(1+\left[\frac{u^{d-p}}{d-p}\right]_1^x \right)\right)\\ T(x)\in\Theta\left(x^p+x^d\right)$$
If we assume that $d\ge p$ then $$T(x)\in\Theta\left(x^d\right)$$
OLD ANSWER: I haven't checked to make sure the sum you provide is actually what you want, but here it goes anyway: You want to evaluate the sum, which I label $f(n)$: $$f(n)=\sum_{i=0}^{\log_\frac43 (n-1)}\left(\frac54\right)^{2id}$$
This is a finite geometric sum, so we simplify to get it in the form $\sum r^i$:
$$\sum_{i=0}^{\log_\frac43 (n-1)}\left(\frac{5^{2d}}{4^{2d}}\right)^{i}= \sum_{i=0}^{\log_\frac43 (n-1)}\left(\frac{25^{d}}{16^{d}}\right)^{i}$$
Using the formula $\sum_{i=0}^nr^i=\frac{1-r^{n+1}}{1-r}$ we obtain:
$$\frac{1-\left(\frac{25^d}{16^d}\right)^{\log_{\frac43}(n-1)+1}}{1-\frac{25^d}{16^d}}= \frac{1-\frac{25^d}{16^d}\left(\frac54\right)^{2d\log_{\frac43}(n-1)}}{\frac{16^d-25^d}{16^d}}= \frac{16^d-25^d\left(\frac54\right)^{2d\log_{\frac43}(n-1)}}{16^d-25^d}$$
Now this is a very ugly expression, but I'm assuming you want to find a function $g(n)$ such that $f(n)=O(g(n))$, given the Computer Science nature of your problem.
$$f(n)=O\left(\left(\frac54\right)^{\frac{2d\log_\frac54(n-1)}{\log_\frac54{\frac43}}}\right) = O\left((n-1)^\frac{2d}{\log_\frac54\frac43}\right)= O\left(n^\frac{2d}{\log_\frac54\frac43}\right)\approx O\left(n^{1.55132\,d}\right)$$