Any roll of a 1, 2, or 3 results in the player winning the amount of the die roll, in dollars, and the termination of the game.
Any roll of a 4 results in the player winning $2 and continuation of the game.
Let Y denote the payoff in dollars of the game. Find E[Y ].
My try: $$\begin{align} E[X] &= \sum_{k=0}^{\infty} \left(2k +\frac{1}{4} +\frac{2}{4} +\frac{3}{4}\right)\frac{1}{4^k}\\ &= 2\sum_{k=0}^{\infty}k\frac{1}{4^k} + \frac{3}{2}\sum_{k=0}^{\infty}\frac{1}{4^k}\\ &= 2\frac{4}{3}\frac{1}{4}\sum_{k=1}^{\infty}k\frac{1}{4^{k-1}}\frac{3}{4}+\frac{3}{2}\frac{4}{3}\\ &= \frac{2}{3}\frac{4}{3} + 2\\ &= \frac{26}{9}. \end{align}$$
The right answer is $\frac{8}{3}.$ What is wrong with my approach?
We can separately count the expected payoff from all rolls of $1$, $2$, or $3$ and the expected payoff from all rolls of $4$.
For the first part we have $$ \sum_{k=0}^\infty \left(\frac14 + \frac24 + \frac34\right)\frac{1}{4^k} = 2. $$
This part of your calculation is correct.
Alternative method: exactly one of these rolls will occur, it will be the last roll, and the last roll has equal probability ($1/3$) to be each of the three possible outcomes, so the expected payoff from these rolls is $$\frac13 + \frac23 + \frac33 = 2.$$
For the second part, you say that if you roll $4$ exactly $k$ times before the end of the game then the payoff from $4$s will be $2k.$
The probability to roll $4$ exactly $k$ times before the end of the game is $$ \frac34 \cdot \frac{1}{4^k}.$$ The expected payoff therefore is $$ \sum_{k=0}^\infty 2k \cdot \frac34 \cdot \frac{1}{4^k}. $$
You did not include the factor of $\frac34$ in your calculation. That is how you got a result greater than the correct result.