In general, we know that the higher direct images of finite morphisms are zero.
Now, suppose we have a morphism $f: S\rightarrow B$ which is finite over a Zariski open subset (say $Z$) of the projective variety $B$. But, the fibers over the points of the complement of $Z$ are not finite.
In the above situation can we still say $Rf_* = f_*$?
Let $f: X \to \mathbb A^2$ be the blow-up of the origin. $f$ is birational, hence satisfies your assumptions. Then $f_* = \Gamma(X,-)$. Clearly this functor is not exact, because $X$ is not affine (Serre's criterion for affine schemes). Thus there are non-zero higher direct images $R^if_*\mathcal F$ for some coherent sheaf $\mathcal F$ on $X$.
Since you want $f$ to map into something projective, just compose with the inclusion $U=\mathbb A^2 \hookrightarrow \mathbb P^2$. Then $\mathbb f_*(-)_{|U} = \Gamma(X,-)$ is not exact, hence $f_*$ is not exact.