Consider the symmetric matrix Riccati differential equation :
$$
P'(t) = Q + X^\intercal P(t) + P(t) X + P(t)UP(t)
$$
Where everything is symmetric and the coefficients are constant (independent of time). The quadratic term U and the term Q are not definite, instead have a mix of positive and negative eigenvalues (say half half), in different bases.
If it can help, U can be written as :
$$
U = -U_1+\gamma U_2
$$
where $U_1$ is symmetric singular and $U_2$ symmetric semidefinite positive.
Is it possible to tackle the issue of finding conditions for boundedness of P ?
This answer does not give a complete answer, but hopefully does help you gain some insights. Namely, it is possible to use a coordinate transformation in which the dynamics is linear, which might be easier to analyse. Namely, if $\bar{P}$ is a stationary solution to
$$ P'(t) = Q + X^\top P(t) + P(t)\,X + P(t)\,U\,P(t), \tag{1} $$
so
$$ Q + X^\top \bar{P} + \bar{P}\,X + \bar{P}\,U\,\bar{P} = 0, \tag{2} $$
then the transformed coordinates can be obtained using $P(t) = \bar{P} + Y^{-1}(t)$, which is equivalent to
$$ Y(t) = (P(t) - \bar{P})^{-1}. \tag{4} $$
By using $(2)$ the dynamics of $Y(t)$ can be shown to simplify to
$$ Y'(t) = -\left(U + Y(t)\,(X^\top + \bar{P}\,U) + (X + U\,\bar{P})\,Y(t)\right). \tag{5} $$
By using vectorization and the Kronecker product this can be written as the following system of linear first order ordinary differential equations in $x(t) = \text{vec}(Y(t))$
$$ x'(t) = M\,x(t) - \text{vec}(U), \tag{6} $$
$$ M = -\left((X + U\,\bar{P}^\top) \otimes I + I \otimes (X + U\,\bar{P})\right), \tag{7} $$
with $I$ the identity matrix of the same size as $P(t)$.
Solutions to $(2)$ can be found using the eigendecomposition of the following matrix
$$ Z = \begin{bmatrix} X & U \\ -Q & -X^\top \end{bmatrix} = V\,\Lambda\,V^{-1}, \tag{8} $$
with $\Lambda$ a diagonal matrix containing the eigenvalues of $Z$ and $V$ containing the corresponding eigenvectors. By defining $W$ as a matrix constructed from any selection of half of the columns of $V$, with
$$ W = \begin{bmatrix} W_1 \\ W_2 \end{bmatrix}, \tag{9} $$
where $W_1$ and $W_2$ are of the same size as $P(t)$. Then a solution to $(2)$ can be found using
$$ \bar{P} = W_2\,W_1^{-1}. \tag{10} $$
It can be noted that $V$ always has an even number of columns because the dimensions of $(8)$ are double that of $P(t)$.
In order for $P(t)$ to remain bounded requires that $Y(t)$ remains nonsingular, which I don't think can be shown easily, but the fact that it has linear dynamics hopefully makes it easier. Two things can be noted that if $M$ is Hurwitz $\lim_{t\to\infty} x(t) = M^{-1}\,\text{vec}(U)$ or if $M$ has at least one eigenvalue with positive real part $\lim_{t\to\infty} \|x(t)\| \to \infty$ for almost all initial condition with $x(t)$ dominated by the eigenmode of $M$ with the eigenvalue which has the largest positive real part. If transforming $M^{-1}\,\text{vec}(U)$ or the eigenvector of the dominant mode back into a square matrix is nonsingular implies that $\lim_{t\to\infty} Y^{-1}(t)$ should remain bounded. If this isn't the case then $\lim_{t\to\infty} P(t)$ should become unbounded. However, I am not sure this is a sufficient condition and there are probably a couple caveats, such as when there are multiple eigenvalues which have equal largest positive real part.