Consider the Hopf bundles $$S^1\rightarrow S^{2n+1}\rightarrow \mathbb{C}P^n$$ and $$S^3\rightarrow S^{4n+3}\rightarrow \mathbb{H}P^n.$$
In this question (and also here), it is shown that for any continuous map $f:X\rightarrow \mathbb{C}P^n$, there is a lift $\tilde{f}:X\rightarrow S^{2n+1}$ iff $f^\ast:H^2(\mathbb{C}P^2)\rightarrow H^2(X)$ is the $0$ map.
I'd like to know a similar characterization for the quaternionic Hopf bundles.
Suppose $f:X\rightarrow \mathbb{H}P^n$ is any continuous function. Is there a nice characterization of when there is a lift $\tilde{f}:X\rightarrow S^{4n+3}$?
The proof for $\mathbb{C}P^n$ relies on the fact that $\mathbb{C}P^\infty$ is a $K(\mathbb{Z},2)$, and so a map to $\mathbb{C}P^\infty$ is homotopically trivial iff the induced map on $H^2$ is $0$. This part of the proof fails for $\mathbb{H}P^n$.
Now, a necessary condition that $f$ have a lift is that $f^\ast$ be the $0$ map on $H^4$. But this condition is not sufficient, as the following example demonstrates:
By the long exact sequence in homotopy groups for the quaternionic Hopf fibration, we see that $\pi_5(\mathbb{H}P^n)$ surjects onto $\pi_4(S^3)\cong\mathbb{Z}_2$, so long as $n\geq 1$. In particular, $\pi_5(\mathbb{H}P^n)\neq 0$.
So, let $X= S^5$ and let $f:X\rightarrow \mathbb{H}P^n$ be homotopically nontrivial. Then $f^\ast$ is the $0$ map on $H^4$ for trivial reasons, but there is no lift $\tilde{f}$ because $f$ can't factor through a $5$-connected space.
This example shows that it is also necessary that $f$ induce the $0$ map on $\pi_5$. One can similarly argue that $f$ must induce the $0$ map on $\pi_6$. But this is as far as I can get.
Thank you!
The map $S^{4n+3} \to \mathbb H P^n$ realizes the source as an $SU(2)$-bundle over the target. By the universal property of pull-backs (fibre products), a morphism $f: X \to \mathbb H P^n$ lifts to a morphism $\tilde{f}: X \to S^{4n + 3}$ if and only if the pulled-back bundle $f^* S^{4n+3}$ (an $SU(2)$-bundle over $X$) admits a section.
So the question seems to break into two parts:
(a) what are the invariants that classify an $SU(2)$-bundle; and
(b) what are these invariants in the particular case of $S^{4n+3}$ over $\mathbb H P^n$.
Now the answer to these questions is at least partly related to Chern classes. If $f^* S^{4n+3}$ is trivial, then $f^*$ should at least kill the Chern classes of $S^{4n+3} \to \mathbb H P^n$. In general, for a rank $2$ bundle, there are two Chern classes, $c_1 \in H^2$ and $c_2 \in H^4$. When the bundle has structure group $SU(2)$, the class $c_1$ vanishes, and so the only interesting Chern class is $c_2 \in H^4$.
This fits well with the case of $\mathbb H P^n$, which has vanishing $H^2$, but whose $H^4$ is one-dimensional, generated by the $c_2$ of the Hopf bundle (I think).
So far, this analysis is completely in line with the corresponding $\mathbb C P^n$ case. But there is a key difference between the two:
For $S^1$-bundles, the first Chern class $c_1$ is a complete invariant (if we work with integral cohomology), so if it vanishes, the $S^1$-bundle is trivial. So in the case of a morphism $f: X \to \mathbb C P^n$, the triviality of $f^* H^2$ is not only necessary, but is also sufficient, for $f$ to lift to $\tilde{f}: X \to S^{2n+1}$.
For $SU(2)$-bundles, the second Chern class $c_2$ is not a complete invariant. My understanding (though I could be mistaken) is that it is a complete rational invariant. But there are additional torsion invariants, such as the class in $\pi_5$ that you found.
In fact, we have that $BSU(2) = \mathbb HP^{\infty}$, with $S^{\infty} \to \mathbb H P^{\infty}$ being the universal $SU(2)$-bundle. (See e.g. the discussion here on p.81.) Now $S^{\infty}$ is contractible, and so we see that $\pi_i(\mathbb H P^{\infty}) = \pi_{i-1}(S^3).$ In particular, if we work with rational homotopy, since $S^3$ has only one non-vanishing rational homotopy group, in degree $3$, we see that $\mathbb H P^{\infty}$ has only one non-vanishing rational homotopy group, in degree $4$. (And its generator is $c_2$ of the universal bundle; so I guess my suggestion above that $c_2$ is a complete rational invariant is correct.)
But $S^3$ has lots of torsion higher homotopy, and hence so does $\mathbb H P^{\infty}$.
At this point, I have to stop, since I'm not sure how to go on, other than noting certain explicit torsion obstructions, as in the OP.
Added: I think we are literally in the context of this wikipedia discussion about obstructions to lifting sections of principal bundles.
The class $c_2 \in H^4$ is the first obstruction, coming from $\pi_3(S^3)$. The next obstruction (if this one vanishes) will be a class in $H^5$ with coefficients in $\pi_4(S^3) = \mathbb Z/2;$ for the map $S^5 \to \mathbb H P^n$ that you discussed, this obstruction will be non-zero.
Since $S^3$ has many (!) other non-vanishing homotopy groups, there will be higher obstructions too (at least as $n$ grows).
For what it's worth, I think that the problem of computing these higher obstructions is also related to the problem of computing a Postinikov tower for $\mathbb H P^{\infty}$.