Given the following Moment-generating function:
$$M_Y(s)=\alpha^6(0.1+2e^s+0.1e^{4s}+0.4e^{7s})^6$$
I want to find the value of $\alpha$, I found two values one positive and one negative using the fact that $M_Y(0)=1$ But how can I decide which $\alpha$ is correct?
As J.G. commented, there's no way to distinguish between the positive and negative candidates for $\alpha$ given the information in the question. This is analogous to: "If $x^2 = 9$, what is $x$?" I suspect there's a bit more in the question that would lead you to pick the positive choice for $\alpha$; for instance, it may be that your m.g.f. corresponds to the sum of $6$ i.i.d. variables, each of which has the m.g.f. $\alpha(0.1 + 2e^s + 0.1e^{4s} + 0.4e^{7s})$.
For a hint on calculating $\mathbb P(X = 11)$: note that when an m.g.f. is presented in this form, it's essentially laying out the probability function of a discrete random variable for you. Consider a random variable $Y$ that takes on the values $7, 8, 9$ with respective probabilities $1/6$, $2/6$, $3/6$; note that such a random variable would have m.g.f. $$\mathbb E[e^{t Y}] = \frac 1 6 \cdot e^{t \cdot 7} + \frac 2 6 \cdot e^{t \cdot 8} + \frac 3 6 \cdot e^{t \cdot 9}$$ and consider what your m.g.f. would reveal if you expanded the $()^6$ term. You may also find the "sum of i.i.d. variables" perspective" I suggested above to be useful.