I have been studying homotopy theory, and I feel very uneasy to handle the lower homotopy groups (sets).
for example, we often use homotopy exact sequences of pairs and the five lemma to say that some homotopy groups are isomorphic. but it seems that the five lemma doesn't hold in the general setting of pointed sets, so I always spend a lot of time to check it (I think we have to use the arbitrarily of base points and chase the paths and points concretely).
More serious problem for me is that many textbooks on homotopy theory do not pay a lot of attention to the definition around relative $\pi_0$ sets. They define a quasifibration as "a continuous map $p:E\to B $ such that $p_\ast: \pi_q(E, p^{-1}(b))\to\pi_q(B)$ is an isomorphism for every choice of base points and for every $q\geq 0$" without defining the relative $\pi_0$ sets (they also use the five lemma for the long exact sequences of triples down to $\pi_0$). Another example is the definition of a map of pairs $f:(X,A)\to (Y,B)$ to be an $n$-equivalence, though some textbooks include the condition that $f_\ast ^{-1}( \mathrm{Im}(\pi_0 B\to \pi_0 Y))=\mathrm{Im}(\pi_0 A\to \pi_0 X)$.
When it comes to the triad homotopy groups, things get even worse.
What is the appropriate definition of relative $\pi_0$ sets (and absolute $\pi_{-1}$ sets)? Is there any general "prescription" for handling these subtle situations around the lower homotopy sets? Is there good references for these problems?
This area can be handled by using fibrations of groupoids, which form a usable algebraic model of what is going on. They were defined in the paper
R.Brown, "Fibrations of groupoids" J. Algebra, 15 (1970) 103-132 pdf
and an account is given in the book (T&G) Topology and Groupoids, (2006), as it was in the 1989 (differently titled) edition. A morphism $p: E \to B$ of groupoids is called a fibration if for each $x \in Ob(E)$ and $b :p(x) \to y$ in $B$ there is an $e$ in $E$, starting at $x$, such that $p(e)=b$.
Here is a version of 7.2.9 of that book.
Let $p : E \to B$ be a fibration of groupoids. Let $x$ be an object of $E$, and let $F_{x}$ be the fibre $p^{-1}[0_{y}]$ of $p$ over $y = px$ . Then there is a sequence of maps of three groups and three pointed sets $$ F_{x}(x)\stackrel{i'}{\longrightarrow} E(x)\stackrel{p'}{\longrightarrow} B(y) \stackrel{\partial}{\longrightarrow} \pi_{0} F_{x} \stackrel{i_{*}}{\longrightarrow} \pi_{0}E \stackrel{p_{*}}{\longrightarrow} \pi_{0}B $$ in which $i'$ is the restriction of the inclusion $i : F_{x} \to E$, $p'$ is the restriction of $p$, $i_{*}$ and $p_{*}$ are the induced maps, and $\partial$ is given by $\partial \alpha = \alpha_{\sharp}\bar{x}$, where $\bar{x}$ denotes the component of $x$ in $F_{x}$ and $\alpha_{\sharp}$ is given by an operation of $B(px)$ on $\pi_{0}F_{x}$. The above sequence has the following properties:
$i'$ is injective and $i' [F_{x}(x)] = p'^{-1}[0_{y}]$;
$\partial \alpha = \partial \beta$ if and only if there is a $\gamma \in E(x)$ such that $p' \gamma = - \beta + \alpha$;
if $\bar{u}$ denotes the component in $F_{x}$ of an object $u$ of $F_{x}$, then $i_{*}\bar{u} = i_{*}\bar{v}$ if and only if there is an $\alpha \in B(y)$ such that $\alpha_{\sharp}\bar{u} = \bar{v}$;
if $\hat{y}$ denotes the component of $y$ in $B$ then $$ i_{*}[\pi_{0}F_{x}] = p^{-1}_{*}[\hat{y}]. $$
The development of the operation $\alpha_\sharp$ referred to above is related to operations used in the theory of covering spaces, and seems (IMHO) also to be a convenient way of constructing useful operations of groupoids on certain homotopy sets, e.g. fundamental groupoids on higher homotopy groups.
The above family of exact sequences can then be applied to topological and other situations. There is also a "Mayer-Vietoris" type sequence, applied in Chapter 10 of T&G to pullbacks of covering spaces.
The paper referred to also discusses 5-lemma type arguments in its Theorem 4.9.
Google scholar will give you more references.
Hope that helps!