This is from Hirsch/Smale/Devaney's textbook "Differential Equations and Introduction to Chaos" Page 199
Theorem. (Lasalle’s Invariance Principle)
Let $X^∗$ be an equilibrium point for $X^\prime = F(X)$ and let $L : U → R$ be a Liapunov function for $X^∗$ , where $U$ is an open set containing $X^∗$ . Let $P ⊂ U$ be a neighborhood of $X^∗$ that is closed. Suppose that $P$ is positively invariant and that there is no entire solution in $P − X^∗$ on which $L$ is constant. Then $X^∗$ is asymptotically stable, and $P$ is contained in the basin of attraction of $X^∗$
I am struggling with the meaning of "no entire solution on which $L$ is constant" part. It seems to be a terminology that is very specific to this theorem and found nowhere else (even in other books). I don't intuitively understand why this is required.
For example, consider the famous Pendulum example,
$$\dot x_1 = x_2$$
$$\dot x_2 = -a\sin(x_1) - bx_2$$
We wish to look at the stability of the equilibrium at $0$. They showed that you can have a Liapunov function, $L(x) = a(1-\cos(x_1)) + x_2^2/2$ and proved $\dot L(x) = 0 \implies x_2 = 0, x_1 \in [0, 2\pi]$ (which is a segment of the $x$-axis)
Then if we are to apply this theorem, we can pick $U$ to be the entire real line, and $P$ be the line-segment $[0, 2\pi]$. Since $\dot L = 0$ on $P$ and $P$ is a line-segment, therefore $P$ is closed and positively invariant. How do we proceed to prove the tricky statement there are no "entire solution in $P \backslash \{0\}$ that on which $L$ is constant?
To do this it seems we need to prove that since any $\{x_2 = 0, x_1 \in (0, 2\pi]\}$ could be a (constant) solution in $P \backslash \{0\}$ on which $L$ is for sure constant, therefore we need to eliminate all $\{x_2 = 0, x_1 \in (0, 2\pi]\}$ as being the solution of this system. Is this right?
It is not necessarily required; though it emphasizes one aspect of the LaSalle principle they want to emphasize, namely, that it can be used to find subsets in the basin of attraction of the equilibrium point in question. In general, roughly speaking the zero set of the trajectory-wise derivative $\dot{L}$ (which is the Lie derivative of $L$ w/r/t $F$ in the statement of the theorem) catches the accumulation points of positive-time trajectories (this is essentially the version of the LaSalle principle stated in the notes you give a link to).
If the zero set of $\dot{L}$ contains no entire trajectory other than the trajectory of the equilibrium point in question, then positive-time trajectories ought to be caught by said equilibrium point, that is, they ought to be in the basin of attraction.
For the pendulum example, first note that $U$ ought to be two dimensional. Likewise it would be better to specify a two dimensional subset as $P$ (if I understand correctly you are only specifying the position variable $x_1$, though this is not quite clear). In the Hirsch-Smale-Devaney book they look at sets of the form
$$P_{c,\epsilon}=\{(x_1,x_2)\,|\, L(x_1,x_2)\leq c\text{ and } |x_1|\leq\pi-\epsilon\},$$
where $c>0$ and $\epsilon>0$ are small enough (and without loss of generality $a=1$). Here the second condition is so as to not consider the other equilibrium point. Also note that $L$ is the energy of the system, so it is natural to consider its sublevel sets. It is also natural due to the conservation of energy, as we need $P$ to be invariant (only in the positive time direction, but still). Here is a humble graph: https://www.desmos.com/calculator/hxbflcnl1t .
To show that such a set does not contain any entire trajectory other than that of $0=(0,0)$, suppose otherwise, and say e.g. the trajectory of $x=(x_1,x_2)\in P_{c,\epsilon}$ stays in $P_{c,\epsilon}$ for all time. then for all time $t$ we have $0=\dot{L}(x(t))=-b x_2(t)^2$, whence $x_1(t)=x_1(0)$ is constant, which can only happen if $x_1(0)=0$, that is, $x$ is the equilibrium point $0$. (The details of this argument are well written in the aforementioned book.)
So the moral is that the compact sets $P$ ought to be chosen in such a way that it becomes easy to verify that they don't contain entire nonequilibrium trajectories along which the time derivative of the Lyapunov function vanishes.
In case there is any confusion as to what "entire solution in $P\setminus\{X^\ast\}$ on which $L$ is constant" means, it means a solution $X(t)$ of the differential equation $X'=F(X)$ (so $\forall t\in\mathbb{R}: X'(t)=F(X(t))$) stays in $P\setminus\{X^\ast\}$ (so $\forall t\in\mathbb{R}: X(t)\in P\setminus\{X^\ast\}$) and the single-variable real valued function $t\mapsto L(X(t))$ is constant. They are implicitly assuming that $F$ is $C^1$, so any solution to the ODE at hand is defined for all time $t$, and $P$ is a priori required to be positively invariant (so if $Y(t)$ is an anonymous solution with $Y(0)\in P$, then $\forall t\in\mathbb{R}_{\geq0}: Y(t)\in P$); the point is the extra condition of not being able to leave the set in backward time also.