I am reading the article called Representations of $U(3)$ in $U(N)$ and there is an example $U(6)$ input irreducible representation $[2,1,1,1,1,0]$, which is then shown to be reduced into $U(3)$ irreps $[6,4,2]$ and $[5,4,3]$. I am new to group theory, but as far as I understand a representation of a group is a homomorphism into a group of matrices with matrix multiplication operation. However, if $U(N)$ itself is a group of matrices, so why to define its representation?
Moreover, what I do not understand is that representation of particular $U(6)$ element should be a matrix, while in the article it is in fact a vector, and the same holds for resulting $U(3)$ irreps.
Rationale: I am a C++ programmer working on the optimization and parallelization of the code that implements the above described reduction. And, I would like to understand the math inside at least a bit, though haven't studied group theory before.
If you have an $n\times n$ matrix, one way to understand it is to decompose $\mathbb{C}^n$ into eigenspaces, if the matrix is diagonalizable. An analogue for a group of invertible matrices is to decompose the vector space into irreducible representations, if the group is semisimple/reductive.
Why define a representation of a matrix group? In applications, representations usually come out of recognizing a symmetry in some pre-existing vector space (for example the solutions to a differential equation), and representation theory aims to be able to give a complete description of any such symmetry that you might come across. Sometimes that existing symmetry is a matrix group, so it is good to be prepared.
A basic example is Fourier analysis of periodic functions. Translation of a periodic function gives $U(1)$ symmetry, and the decomposition of a periodic function along the corresponding irreducible representations is the Fourier transform. (Forgive me for glossing over convergence and such.)
A slightly more involved example is Fourier analysis of functions on the unit sphere in $\mathbb{R}^3$. The sphere has $SO(3)$ symmetry, and the symmetry can be used to decompose such functions into elements of odd-dimensional irreducible representations. Spherical harmonics are a certain basis of these vector spaces and have connections to electron orbitals.