Question Statement:-
Let $a,b,c$ be positive integers and consider all the quadratic equations of the form $ax^2-bx+c=0$ which have two distinct real roots in $(0,1)$. Find the least positive integers $a$ and $b$ for which such a quadratic equation exist.
Attempt at a solution:-
Let $f(x)=ax^2-bx+c$. Now as the roots of $f(x)=0$ lie in $(0,1)$ hence $f(0)\cdot f(1)\gt0$.
$$D\gt0\implies b^2-4ac\gt0$$ $$f(0)=c \\ f(1)=a-b+c$$
As, $a,b,c\in\mathbb{Z}^+$, so $$f(0)\ge1 \implies c\ge1\tag{1}$$ $$f(1)\ge1\implies a-b+c\ge1\tag{2}$$ $$f(0)\cdot f(1)\ge1\implies c(a-b+c)\ge1\implies c^2+(a-b)c-1\ge0\tag{3}$$
Now, for the quadratic inequality in $c$ that we obtained in $(3)$, for it to hold we get
$$D\le0\implies (a-b)^2+4\le0$$ The above inequality doesn't hold $\forall a,b \in \mathbb{Z}^+$.
After arriving at this conclusion I am kind of stuck on what to conclude from the above result or rather am I going in the right direction.
A little push in the right direction in the form of a hint or rather what should be the line of thought for attempting the question. Please don't write the solution, just the line of thought to arrive at the solution would be enough.
After a thinking a lot on the question I was able to come up with a good enough solution.
Let $f(x)=ax^2-bx+c$. Let the roots of the equation $f(x)=0$ be $\alpha$ and $\beta$. Then, $\alpha,\beta\in(0,1)$.
As the roots of the equation $f(x)=0$ are real and distinct, then $b^2\gt4ac$.Also, due to the roots lying in $(0,1)$, hence $f(0)\gt0$, $f(1)\gt0$ and $f(0)f(1)\gt0$
Now, $$f(0)=c\ge1\tag{1}$$ $$f(1)=a-b+c\ge1\tag{2}$$ $$\therefore \qquad f(0)\cdot f(1)\ge1\qquad\qquad\tag{3}$$
If $\alpha$ and $\beta$ are the zeroes of $f(x)$, then $f(x)=a(x-\alpha)(x-\beta)$ $$\therefore f(0)\cdot f(1)=a\alpha(1-\alpha)a\beta(1-\beta)\ge 1$$
Using A.M-G.M inequality we get, $$\dfrac{b+(1-b)}{2}\ge\sqrt{b(1-b)}\implies b(1-b)\le\dfrac{1}{4}$$ equality occurs when $b=\dfrac{1}{2}$ $$\therefore \qquad\qquad\alpha(1-\alpha)\le\dfrac{1}{4}\qquad\qquad \beta(1-\beta)\le\dfrac{1}{4}$$
In both the cases equality occurs when $\alpha=\beta=\dfrac{1}{2}$, but as $\alpha\neq\beta$
So, $$\alpha(1-\alpha)\beta(1-\beta)\lt\dfrac{1}{16}\implies a\alpha(1-\alpha)a\beta(1-\beta)\lt\dfrac{a^2}{16}\\ \implies \dfrac{a^2}{16}\gt f(0)\cdot f(1)\implies |a|\gt4$$
$$\therefore a_{min}=5$$
Also, $$\because \qquad\qquad\alpha\cdot\beta\lt1\implies c\lt a_{min}\qquad\qquad\tag{4}$$
$$(2)\implies a+c\ge1+b\implies a_{min}+c-1\ge b\implies 4+c\ge b \\ \implies c^2+8c+16\ge b^2\implies c^2+8c+16 \gt 4a_{min}c\\ \implies c^2-12c+16\gt0\implies c\in(0,6-2\sqrt5)\cup(6+2\sqrt5,\infty)$$
But, as $c\lt a_{min}$, hence $c=1$
Now, from $$b^2\gt 4a_{min}c\implies b^2\gt20$$ Hence, $b_{min}=5$
So, the value of least positive integers $a$,$b$ and $c$ for which $f(x)$ has roots b/w $0$ and $1$ is $(a,b,c)=(5,5,1)$
Other answers are still welcome.