I am studying about Hydraulic and this figure is about "Geometric Properties of Some Common Prismatic Channels"
It says the cross-section area of flow normal to the direction of flow can be calculated from this formula.
I don't understand how the formula ($\frac{1}{8}(\phi-\sin\phi)D^2)$ calculate the area of triangle. and also the book doesn't say anything about $\phi$ is based on radian or degree(I guess it is radian not sure why). I am confused.
The angle is in radians. Also, the area is not just of the shaded part but of the whole cross-section of the liquid (shaded triangle and the sector below it).
Let $R$ be the radius of the circle, $D=2R$.
The area of the sector of the circle "below" the shaded triangle is given by $\frac{1}{2}R^2\phi$. This is really $R^2\pi\times\frac{\phi}{2\pi}$ - i.e. the fraction of the area of the whole circle that corresponds to the fraction $\phi$ of the "full" angle $2\pi$ radians.
The area of the shaded triangle, as per sine formula, comes up as $\frac{1}{2}R^2\sin(2\pi-\phi)=-\frac{1}{2}R^2\sin\phi$.
(note that on your picture $\pi\le\phi\le 2\pi$, so $\sin\phi$ is actually negative, and so the expression above comes out as positive).
Add them together, and you get that the total area is:
$\frac{1}{2}R^2\phi-\frac{1}{2}R^2\sin\phi=\frac{1}{2}(\phi-\sin\phi)R^2$.
Now, substituting $R=\frac{1}{2}D$, you get the desired formula.