A theorem of Deligne asserts that to cusp forms with Euler product, there is for each prime $\ell$ a Galois representation of $G(K_{\ell}/\mathbb{Q})$, where $K_{\ell}$ is the maximal abelian extension of $\mathbb{Q}$ ramified only at $\ell$, satisfying
$$ \det \circ \rho_{\ell} = \chi_{\ell}^{k-1} $$
where $\chi_{\ell}$ is the natural map to $Gal(K_{\ell}^{ab}/\mathbb{Q}) \simeq \mathbb{Z}_{\ell}^*$. This Galois representation is "exceptional at $\ell$" if it is not surjective.
If a Galois representation is exceptional, then its image in $GL_2(\mathbb{F}_{\ell})$ must either be (i) contained in a Borel subgroup, (ii) contained in the normalizer of a Cartan but not the Cartan itself, or (iii) isomorphic to $S_4$ under the map to $PGL_2(\mathbb{F}_{\ell})$.
I understand how one might prove that a Galois representation is not exceptional for a prime $\ell$. Exceptionality imposes conditions, e.g. on the coefficients, and there are results that can immediately reduce one to checking a finite set of primes for a given modular form.
In addition, showing that a Galois representation is exceptional of type (i) or (ii) amounts of an equality of two modular forms, which can be checked with a finite computation.
How can one prove that a Galois representation is exceptional of type (iii)?
This paper by Andrew Sutherland may help: Computing images of Galois representations attached to elliptic curves
Images of Galois representation of elliptic curves without complex multiplication are computed, and the similar method may be applied to modular forms.