In this post there is shown that for a standard Brownian motion $\mathbb{E}[\tau^p]<\infty$ for all $p \geq 1$, where
\begin{align}
\tau = \inf\{t \geq 0 : B_t = \alpha \ \ \text{or}\ \ B_t=-\beta\}.
\end{align}
Now, $\mathbb{E}[B_\tau]$ can be written as
\begin{align}
&= \alpha P(B_\tau = \alpha) - \beta P(B_\tau = -\beta) \\
&= \alpha P(B_\tau = \alpha) - \beta (1 - P(B_\tau = \alpha)) \\
&= (\alpha + \beta) P(B_\tau = \alpha) - \beta.
\end{align}
Since, $B_\tau$ is a martingale, $B_{\tau \wedge t}$ is a martingale as well. So,
\begin{align}
\mathbb{E}[B_{\tau \wedge t}\mid \mathcal{F}_0] = B_{\tau \wedge 0} = B_0 = 0.
\end{align}
Therefore,
\begin{align}
\mathbb{E}[B_{\tau \wedge t}] = 0.
\end{align}
Since $| B_{\tau \wedge t} | \leq \max(\alpha, \beta)$ (why?), by the Dominate Convergence Theorem we find that $\mathbb{E}[B_{\tau}] = 0$. Hence, we can conclude that $\mathbb{P}(\{B_\tau=\alpha\})=\frac{\beta}{\alpha+\beta}$.
Likewise, since $B_\tau^2 - \tau$ is a martingale, $B_{\tau \wedge t}^2 - \tau$ is a martingale. Why is $| B_{\tau \wedge t}^2 | + \tau \leq \max(\alpha^2, \beta^2) + \tau$? And how to conclude from this, by the DCT, that $\mathbb{E}[B_{\tau }^2]=\alpha \beta $?
Furthermore, I am wondering how to determine that , $\mathbb{E}[M_{\tau}] = 0$ for $M_t = B_t^2 - t^2$ for $t\geq 0$ and how to conclude that $\mathbb{E}[\tau] = \alpha \beta$?
Stopping times may be seen as hitting times of your process $(B_t)_{t \geq 0}$ into a set $A=\{\alpha,-\beta\}$. That is, our process stops whenever $B_t=\alpha$ or $B_t=-\beta$.
Think about a particle moving up or down, on a plot where $B_t$ is on the y-axis and $t$ is on the x-axis. Also, wlg suppose $\alpha>0$ and $-\beta<0$.
So, as times goes by, our particle will move up or down on this plot and $\tau = \inf\{t \geq 0 : B_t = \alpha \ \ \text{or}\ \ B_t=-\beta\}$ means that we count the time our particle hits the line of $B_t = \alpha$ or $B_t=-\beta$ (call this the bounds of our particle's movement).
Since $\tau \wedge t$ is a bounded stopping time (do you know why this is the case?), we "know" (almost surely) that our particle will hit either one of the bounds in a finite time.
Now, imagine the case where our particle only goes up. By the above argument, the maximum value that $B_{\tau \wedge t}$ can reach at time $\tau \wedge t$ is $\alpha$. Then, $B_{\tau \wedge t} \leq \alpha$.
The same way, if it only goes down, the minimum value it reaches is $-\beta$. And, $B_{\tau \wedge t} \geq -\beta$.
Thus, $| B_{\tau \wedge t} | \leq \max(\alpha, \beta)$.
Same idea as the above.
You said that "by the Dominate Convergence Theorem we find that $\mathbb{E}[B_{\tau}] = 0$. Hence, we can conclude that $\mathbb{P}(\{B_\tau=\alpha\})=\frac{\beta}{\alpha+\beta}$".
Therefore, $$\mathbb{E}[B_{\tau }^2] = \alpha^2 \frac{\beta}{\alpha+\beta} + \beta^2 \frac{\alpha}{\alpha+\beta} = \alpha \beta$$
Is the definition of $M_t$ correct? Are you sure it's not $M_t = B_t^2 - t$?
Considering $M_t = B_t^2 - t$ and since $\mathbb{E}[B_\tau^2 - \tau] = \mathbb{E}[M_{\tau}] = 0$. Then, $$\mathbb{E}[B_\tau^2] - \mathbb{E}[\tau]=0 \implies \mathbb{E}[B_\tau^2] = \mathbb{E}[\tau].$$
And we have seen that $\mathbb{E}[B_\tau^2] = \alpha \beta$. Thus, $\mathbb{E}[\tau] = \alpha \beta$.