I would like to understand and if possible classify all vector subbundles of a fixed given one $E$ on the complex projective line, in the holomorphic category. The Grothendieck decomposition theorem says that over $\mathbb{CP}^1$ $$ E \simeq \bigoplus_{i=1}^{r} \mathcal{O}(k_i) $$ where $r$ is the rank of $E$.
This is partially answered in this question but unfortunately the general case is never covered there.
What are the general subtleties that arise (as the answer there suggests) in general? What kind of (probably cohomological) techniques could be used to tackle this problem?
To start somewhere, this is trivial for line bundles: $L' \subset L$ as holomorphic lines bundles implies that either $L' = 0$ or $L' = L$.
This can be done directly from writing down such a map.
Consider a map $\bigoplus_{j=1}^s O(d_j) \to \bigoplus_{i=1}^r O(k_i)$ where $s<r$. This is given by an $r \times s$ matrix of forms $A = (F_{ij})$ where $\deg F_{ij} = k_i - d_j$. Assume the $k$’s and $d$’s are sorted in weakly decreasing order, so the entries of the matrix increase in degree from left to right and decrease in degree from top to bottom. In particular the entries constrained to be zero for degree reasons form a region that is downwards- and leftwards-closed.
The matrix drops rank at a point $x \in \mathbb{P^1}$ if and only if all the maximal minors are zero at $x$. So, a necessary condition is that the main diagonal is nonvanishing: $k_i - d_i \ge 0$ for $i=1, \ldots, s$. Otherwise all the minors are identically 0.
If this condition holds, that still only implies the map is generically injective (i.e. gives a locally free subsheaf), not injective on every fiber.
It turns out the requirement depends on the next diagonal below the main diagonal. I’ll assume $k_1 - d_1 > 0$, since otherwise the top left entry is constant, meaning $O(d_1)$ maps isomorphically to $O(k_1)$ and the entire map is injective iff the remaining map is (with those two summands deleted).
Then the $(i+1,i)$ diagonal has to be nonvanishing: $k_{i+1} - d_i \ge 0$ for all $i$. The reason is as follows: if the $(i+1,i)$ entry vanishes for degree reasons, let $D$ be the determinant of the top left $i \times i$ matrix; this is nonconstant since $d_1 < k_1$. Then setting $D=0$ gives a a nonempty subset of $\mathbb{P}^1$ on which the matrix drops rank as columns $1,\ldots,i$ become linearly dependent.
Conversely, if all the $(i+1,i)$ entries are nonzero, a general choice makes the determinants of the top $s$ rows and of rows $2,\ldots, s+1$ coprime. For example, put powers of $X$ down the main diagonal and powers of $Y$ down the next diagonal, and zeroes elsewhere. (Since this matrix has full rank everywhere on $\mathbb{P^1}$, so does a general choice of matrix.)
Summary: with the $k$’s and $d$’s sorted in weakly decreasing order, there is a locally free subsheaf if and only if $d_i \le k_i$ for $i=1, \ldots, s$, and there is a subbundle if and only if the $d$’s have the form $d_i = k_i$ for $i < s’$ for some $1 \le s’ < s$, then $d_i \le k_{i+1}$ for $i=s’, \ldots, s$.
Example: if $E = O(6) \oplus O(3) \oplus O$, the proper nontrivial subbundles are: