Assume that ellipse $C:\frac{x^2}{a^2}+\frac{y^2}{b^2}=1(a>b>0)$, its left focus is $F$, the line $l$ passing $F$ intersects the ellipse $C$ at $A$ and $B$. The angle of inclination of $l$ is $\frac{\pi}{3}$, and $\overrightarrow{AF}=2\overrightarrow{FB}$.
If $|AB|=\frac{15}{4}$, how to solve the equation of $C$?
On
Fig. 1 :This figure illustrates the 2 answers I give as well as the very interesting answer by @SMM. On the left is figured the directrix line associated with focus $F$. See comments at the end of the first solution.
First method
Let us use the following parameterization of the ellipse :
$$\begin{cases}x=a \cos(t)\\y=b \sin(t)\end{cases}\tag{1}$$
Let
$$\begin{cases}t_1, c_1=\cos(t_1), s_1=\sin(t_1) \ \text{correspond to point } \ A \\t_2, c_2=\cos(t_2), s_2=\sin(t_2) \ \ \text{correspond to point } \ B\end{cases}$$
The following system of 6 equations with 6 unknowns takes into account the different constraints of the problem :
$$\begin{cases}\frac{b(s_1-s_2)}{a(c_1-c_2)}&=&\sqrt{3}& \ \ \ \ (a)\\ a^2(c_2-c_1)^2+b^2(s_2-s_1)^2&=&(\tfrac{15}{4})^2& \ \ \ \ (b)\\ a\frac{c_1+2c_2}{3}&=&-\sqrt{a^2-b^2}& \ \ \ \ (c)\\ 2s_2+s_1&=&0& \ \ \ \ (d)\\ s_1^2+c_1^2&=&1& \ \ \ \ (e)\\ s_2^2+c_2^2&=&1& \ \ \ \ (f)\end{cases}\tag{2}$$
Explanations :
(a) for the slope $\tan(\tfrac{\pi}{3})=\sqrt{3},$
(b) for $\|\overrightarrow{AB}\|^2=(\tfrac{15}{4})^2,$
(c) and (d) for relationship $\tfrac{A+2B}{3}=F$, knowing that the coordinates of this focus are $(c,0)=(-\sqrt{a^2-b^2},0).$
No comment for (e) and (f).
I have used a CAS for solving this system, but it could be done by hand (for example, one can at once get rid of $s_1$ using eq. (1)d).
The solutions of system (1) are :
$$a=3, \ b=\sqrt{5}, \ c_1=-1/4, \ c_2=-7/8, \ s_1 =-\sqrt{15}/4, \ \ s_2=\sqrt{15}/8$$
giving the following equation :
$$\dfrac{x^2}{9}+\dfrac{y^2}{5}=1$$
and the following coordinates for points $A, B, F$:
$$A=(a c_1,b s_1)=(-21/8,-5\sqrt{3}/8), \ \ \ \ B=(a c_2,b s_2)=(-3/4,5\sqrt{3}/4)$$
and $F=(-\sqrt{a^2-b^2},0)=(-2,0).$
Remarks : First, a point of vocabulary : a chord like $AB$ passing through one of the focii is called a focal chord. One can prove that the pole $P$ of a focal chord (intersection of the tangents in $A$ and $B$) is situated on the directrix. More precisely, the directrix associated with $F$ can be described as the locus of poles of all focal chords passing through $F$. Moreover $PF \perp AB$ ; an old reference about all that : https://archive.org/stream/geometricaltreat00drewuoft/geometricaltreat00drewuoft#page/38/mode/1up. This gives :
2nd method of solution :
Let us first recall that the tangent to an ellipse with equation $\dfrac{x^2}{a^2}+\dfrac{y^2}{b^2}=1$ at point $(x_k,y_k)$ has the following equation
$$\dfrac{xx_k}{a^2}+\dfrac{yy_k}{b^2}=1.\tag{3}$$
Let us now write the equation of the ellipse under a form that will be more handy for expressing constraints:
$$ux^2+vy^2=1 \ \ \text{with} \ \ u:=1/a^2, v:=1/b^2.\tag{4}$$
Let us denote by $(x_0,y_0)$ the coordinates of point $P$ (the so-called "pole").
Then, the unknowns, that are different from the unknowns of system (2), verify :
$$\begin{cases} y_1-y_2&=&\sqrt{3}(x_1-x_2)& \ \ \ \ (a)\\ (x_1-x_2)^2+(y_1-y_2)^2&=&225/16& \ \ \ \ (b)\\ (2x_1+x_2)/3&=&f& \ \ \ \ (c)\\ 2y_1+y_2&=&0& \ \ \ \ (d)\\ (x_0-f)(x_1-x_2)+y_0(y_1-y_2)&=&0& \ \ \ \ (e)\\ ux_0x_1+vy_0y_1&=&1& \ \ \ \ (f)\\ ux_0x_2+vy_0y_2&=&1& \ \ \ \ (g)\\ ux_1^2+vy_1^2&=&1& \ \ \ \ (h)\\ ux_2^2+vy_2^2&=&1& \ \ \ \ (i)\end{cases}\tag{5}$$
Explanations :
(a), (b), (c), (d) : same constraints as in system (2).
(e) : orthogonality between PF and AB.
(f) : $P=(x_0,y_0)$ is on the tangent in $A(x_1,y_1)$ (using (3)).
(g) : In the same way, $P=(x_0,y_0)$ is on the tangent in $B(x_2,y_2)$.
(h) and (i) : $A$ and $B$ belong to the ellipse.
System (5), although bigger than system (2) (9 equations with 9 unknowns) is easily solved by a CAS, or even by hand. It gives in particular
$$P(x_0,y_0)=(-9/2,5\sqrt{3}/6).$$
By the assumption we have: $AF=\frac{10}{4}$ and $BF=\frac{5}{4}$.
We have that left focus is $F=(-c,0)$ where $c^2=a^2-b^2$. Consider the left directrix $d:\ x=-\frac{a}{e}$, where $e= \frac{c}{a}$ is the eccentricity, and drop perpendiculars $A',F',B'$ from $A,F,B$ to $d$. The quadrilateral $AA'B'B$ is a trapezoid with the angle $\angle BAA'=60^\circ$, right angles at vertices $A'$ and $B'$, and basis $AA'=\frac{1}{e}AF=\frac{10}{4e}$ and $BB'=\frac{1}{e}BF= \frac{5}{4e}$. Then we easily see (by dropping a perpendicular from $B$ to $AA'$) that $AA'-BB'=AB\cos 60^\circ$, so $\frac{5}{4e}=\frac{15}{4}\cdot \frac{1}{2}$, wherefrom $e=\frac{2}{3}$. Therefore, $AA'=\frac{15}{4}$ and $BB'=\frac{15}{8}$.
Since parallel line $FF'$ to the basis of the trapezoid divides $AB$ in the ratio $2:1$ we have: $FF'=\frac{1}{3}(AA'+2BB')$, i.e. $\frac{a}{e}-c= \frac{1}{3}\cdot\frac{30}{4}= \frac{10}{4}$, i.e. $\frac{a}{e}-ae=a(\frac{3}{2}-\frac{2}{3})=\frac{10}{4}$, wherefrom $a=3$. Now, $c=ae=2$, and $b=\sqrt{a^2-c^2}=\sqrt{5}$.