First of all, I reveal this problem stems from below statement.
a graph $G = (V, E)$ with $n$ vertices is extremal for $K_3$ if it contains "no triangles" and has $\left\lfloor\frac{n^2}{4}\right\rfloor$ edges.
Thm : For every $n$, every extremal $n$-vertex graph for $K_3$ is isomorphic to the graph $K_{a, b}$ with $a=\left\lfloor\frac{n}{2}\right\rfloor$, $b = n -\left\lfloor\frac{n}{2}\right\rfloor$.
So, conversely saying, if the number of edges of any $n$-vertex graph is greater than $\left\lfloor\frac{n^2}{4}\right\rfloor$, We can find a triangle(or triangles), that is, can find at least "one" triangle.
Likewise, if average degree of the n-vertex graph is greater than $\frac{n}{2}$, so the number of edges is the graph is greater than $\frac{n^2}{4}$, thus we can find at least one triangle.
Back to main problem, I can't reach how can I find "different $\frac{1}{10}$ ${n\choose 3}$ triangles" if the average degree of the graph ($n$-vertex) is greater than $\frac{3n}{5}$.
I don't just want to find answers.
If you're willing to help me, Give me hints, please.
thanks for reading.
I have a weaker inequality (that is, I got $\displaystyle \frac1{25}\binom{n}{3}$ instead of $\displaystyle \frac1{10}\binom{n}{3}$). Here is a proof. Maybe somebody can tweak it and get the bound you need.A branch of a graph is a pair $\big(v,\{e,f\}\big)$, where $e$ and $f$ are two distinct edges with the same endpoint $v$ such that $e$ and $f$ are not edges of a triangle. Let $b(G)$ and $t(G)$ denote the number of branches and triangles in a simple graph $G$ on $n$ vertices with $m$ edges.
Observe that $$b(G)+3\,t(G)=\sum_{v\in V(G)}\,\binom{\deg_G(v)}{2}\,,$$ where $V(G)$ is the vertex set of $G$ and $\deg_G(v)$ is the degree of each $v\in V(G)$ in $G$. By, for example, the AM-QM Inequality, we can show that $$\sum_{v\in V(G)}\,\binom{\deg_G(v)}{2}\geq n\,\binom{\frac{1}{n}\,\sum\limits_{v\in V(G)}\,\deg_G(v)}{2}=n\,\binom{\left(\frac{2m}{n}\right)}{2}\,.$$ This proves that $$b(G)+3\,t(G)\geq \frac{n}{2}\,\left(\frac{2m}{n}\right)\,\left(\frac{2m}{n}-1\right)=\frac{m\,(2m-n)}{n}\,.\tag{*}$$ However, it is easy to see that $$b(G)+t(G)\leq \binom{n}{3}\,.$$ Therefore, $$t(G)\geq \frac{1}{2}\,\left(\frac{m\,(2m-n)}{n}-\binom{n}{3}\right)\,.$$ This means $$t(G)\geq \frac{6m(2m-n)-n^2(n-1)(n-2)}{12n}\,.$$ If the average degree is greater than or equal to $\dfrac{3n}{5}$, then $$m\geq \frac{n}{2}\left(\dfrac{3n}{10}\right)=\frac{3n^2}{10}\,.$$ Therefore, $$t(G)\geq \frac{n(n^2+15n-25)}{150}=\frac{(n^2+15n-15)}{25(n-1)(n-2)}\,\binom{n}{3}\,.$$ It can be easily seen that $$\frac{(n^2+15n-15)}{25(n-1)(n-2)}>\frac{1}{25}$$ for all $n\geq 3$. Hence, if $n\geq 3$ and $m\geq \dfrac{3n^2}{10}$, we indeed have $$t(G)>\frac{1}{25}\,\binom{n}{3}\,.$$
We can also count anti-branches and anti-triangles to improve the bound, I think. I am too exhausted to think about them.Edit. Let $b'(G)$ and $t'(G)$ denote the number of branches and triangles, respectively, of the complementary graph $G'$ of $G$ (that is, they count anti-branches and anti-triangles of $G$, respectively). In other words, $b'(G)=b(G')$ and $t'(G)=t(G')$. Write $\deg'_G(v)$ for $\deg_{G'}(v)$ for each $v\in V(G)=V(G')$.
Then, we also have $$b'(G)+3\,t'(G)=\sum_{v\in V(G)}\,\binom{\deg'_G(v)}{2}\,.$$ However, $$\sum_{v\in V(G)}\,\deg'_G(v)=n(n-1)-2m\,.$$ Therefore, $$b'(G)+3\,t'(G)\geq n\,\binom{\frac{1}{n}\,\sum\limits_{v\in V(G)}\,\deg'_G(v)}{2}=n\,\binom{n-1-\frac{2m}{n}}{2}\,.$$ Thus, $$b'(G)+t'(G)\geq \frac{1}{3}\,\big(b'(G)+3\,t'(G)\big)\geq\frac{n}{3}\,\binom{n-1-\frac{2m}{n}}{2}\,.$$ That is $$b'(G)+t'(G)\geq \frac{(n^2-n-2m)(n^2-2n-2m)}{6n}\,.$$ Because $$b(G)+t(G)+b'(G)+t'(G)=\binom{n}{3}\,,$$ we get $$b(G)+t(G)\leq \binom{n}{3}-\frac{(n^2-n-2m)(n^2-2n-2m)}{6n}\,.$$ From (*), we obtain a better bound $$t(G)\geq \frac{1}{2}\,\left(\frac{m\,(2m-n)}{n}-\binom{n}{3}+\frac{(n^2-n-2m)(n^2-2n-2m)}{6n}\right)\,,$$ leading to $$t(G)\geq \frac{m(4m-n^2)}{3n}\,.\tag{#}$$ If $m\geq \dfrac{3n^2}{10}$, then $$t(G)\geq \frac{n^3}{50}=\frac{3n^2}{25(n-1)(n-2)}\,\binom{n}{3}\,.$$ It can easily be seen that $$\frac{3n^2}{25(n-1)(n-2)}>\frac{3}{25}$$ for all $n\geq 3$. Therefore, $$t(G)>\frac{3}{25}\,\binom{n}{3}$$ when $n\geq 3$ and $m\geq \dfrac{3n^2}{10}$.
In fact, for any $\alpha>\dfrac14$, if $n\geq 3$ and $m\geq \alpha\,n^2$, then $$t(G)\geq\frac{\alpha\,(4\alpha-1)}{3}\,n^3>2\alpha\,(4\alpha-1)\,\binom{n}{3}\,.$$ In terms of the average degree $d(G)$ of the graph, we have $$t(G)\geq\frac{d(G)\,\big(2\,d(G)-n\big)\,n}{6} >\frac{d(G)\,\big(2\,d(G)-n\big)}{n^2}\,\binom{n}{3}$$ if $n\geq 3$ and $d(G)>\dfrac{n}2$.
Incidentally, (#) also shows that, if $m> \dfrac{n^2}{4}$, then the graph has a triangle. This is quite an unexpected result because it is sharp, but I thought my bounds were weak.