Finding Quadrants of ellipse from ellipsoid of a Conic section

Question

This is my first post here, hope I won't be giving tough time for you.

I will be giving bit non relevant information here to describe my problem as it may help understand the problem better. I will be talking about irregular or oblique cone ( call it a limit-cone) whose boundaries ($q1, q2, q3$ and $q4$ as shown in Image1) defined by four angles $\theta1$, $\theta2$, $\theta3$ and $\theta$4. Note that this limit-cone is oriented in 3D by a quaternion $Q_{lc}$. Now, lets say I have a vector (from point $p_i$ to a target point $ p_{t}$). How do we determine this vector is inside or outside of limit-cone ?

I'm working on Inverse kinematics algorithm FABRIK: A fast, iterative solver for the Inverse Kinematics problem by Andreas Aristidou . In case if you are unable to access the full paper, please download here.

This algorithm describes an approach for joint constraints in section 4.3 , pseudo-code given in Algorithm 2, 3 in the paper. There are two kinds of constraints discussed: orientational(1 DoF), rotational(2 DoFs). Orientational constraints is easy, but rotational constraint is not. Rotational constraint for a joint is explained using a irregular conic shape.

Figure 4: Irregular cone at Ball-socket joint Pi

---

Figure 5: Rotational and orientational constraints within FABRIK.

The implementation of Algorithm 3 is below for rotational constraints. For visual representation see Fig.5 in the paper.

void Algorithm3(GameObject objN, GameObject objN_1, Vector3 targetPos, Quaternion targetOri, float[] limitAngles)
{
    Quaternion oribtw = Quaternion.Inverse(objN.transform.rotation) * objN_1.transform.rotation;
    Vector3 P1online = objN_1.transform.position; Vector3 P2online = objN.transform.position;
    Vector3 line = P1online - P2online; // direction 
    //project 'targetPos' on 'line' this results in a point 'O'.
    Vector3 pointO = Vector3.Project( (targetPos-P1online), (P1online-P2online) ) + P1online;       
    // calculate distance 'S' between 'O' and 'objN'
    float distanceS = Vector3.Distance(pointO, objN_1);
    float[] limitLengths = new float[4];
    for( int i=0; i< 4; i++){
     limitLengths[i] = S * Math.tan(limitAngles[i]);
    }
// step3.4: Map the target ( rotate and translate) in such a way that 'pointO' is now located at the axis origin and oriented according to the x and y-axis.
// step3.5: Find in which quadrant the target belongs.
// step3.6 ---> step3.14 
}

I could not be able to understand the algorithm from step3.4. Any help in implementation would be really helpful. Thanks in advance.