For some reason I can't get a good hold of those topics (I'm reading Brian C. Hall's Lie Groups, Lie algebras and Representations. So it's matrices only). I'll try to narrow it a bit more:
Irreducibility - I've found only one example of proof showing that representation is irreducible and it was straightforward from the definition: that it doesn't contain non-trivial, invariant subspaces. Is there any other, not too advanced way? Using roots/weights?
Weights - Only one example was given in the book, for the space of $sl(3,\mathbb{C})$. For this space, a special basis was chosen with $H_1 = diag(1,-1,0)$ and $H_2 = diag(0,1,-1)$ and then the weights are $\mu = (m_1,m_2) \in \mathbb{C}$ such that $$\pi(H_1)v = m_1v,$$ $$\pi(H_2)v=m_2v.$$ How do I find a "good" basis in general? The trivial basis seems not to work in that case.
And one last question, about the highest weight - I have found two, possibly equivalent definitions and I'm not sure which one to use:
(Lecture) All $v$ such that $\pi(z)v = v$ and $z \in Z(G)$, when $Z(G)$ is the triangular matrices with ones along the diagonal in $GL_n$.
(Book) By comparison of the weights using positive roots of a representation - $\mu_1 > \mu_2$ iff $\mu_1 - \mu_2 = a\alpha_1 + b\alpha_2$, when $\alpha_1,\alpha_2$ are the positive roots and $a,b \geq 0$.
Thanks in advance.
A) The tools for testing irreducibility depend on the setting. One of the most powerful methods for representations of finite groups is to compute the norm of the character - the representation is irreducible if and only if the norm is 1 (This also works for compact Lie groups, provided you know the character and Haar measure).
The most relevant result here about representations of Lie groups is that the highest weight representations (i.e. those obtained by applying negative roots to the higest weight vector) are irreducible. In practice, if you want to know how a given representation decomposes into irreducibles (and in particular if it's irreducible), you just draw its weights (assuming you can draw in r-dimensional space, where r is the rank:)), and decompose the weight diagram into weight diagrams of the highest weight representations, which are all classified for simple Lie algebras.
B) Note that the weights of a representation, considered as elements of the dual $\mathfrak{h}^*$ of the Cartan subalgebra, do not depend on the basis of $\mathfrak{h}$. However, if you want to draw them in a weight diagram, you have to pick a basis for $\mathfrak{h}$, which chooses the dual basis for $\mathfrak{h}^*$. For example, the Cartan subalgebra of $\mathfrak{sl}(3)$ consists of all 3x3 traceless matrices, so a possible basis is $$\begin{pmatrix}1 & 0 & 0\\0 & -1 & 0\\ 0 & 0 & 0 \end{pmatrix},\quad \begin{pmatrix}1 & 0 & 0\\0 & 0 & 0\\ 0 & 0 & -1 \end{pmatrix} $$ There's nothing like a good or bad basis, although with some choices (when the Killing form is the identity), the weight diagrams will look more symmetric.
C) A highest weight vector is one annihilated by all the positive roots (considered as elements of the Lie algebra). This is just your definition (1) translated from Lie group to Lie algebra setting. To explain your definition (2) from this point of view, note that if a positive root $\alpha$ does not annihilate a given vector with weight $\beta$, the resulting vector will have weight $\alpha+\beta$, leading to a greater value of $\mu$. Thus a vector which maximizes $\mu$ must be annihilated by all positive roots. Note that in a highest weight representation (with a unique highest weight vector), the converse is also true.